ANALYSIS OF OUTREACH
IN THE
RADON AND INDOOR AIR PROGRAMS
Region 10
Policy, Planning and EvaJuation Branch
May, 1993
Background
Region 10's Radon and Indoor Air Program managers requested the Policy,
Planning and Evaluation (PP&E) Branch t...
•••
TABLE OF CONTENTS
I. Executive Summary
Page
4
II. Radon and Indoor Air Pollution 5
III. Bennett Model 8
IV. Explanatio...
EXECUTIVE SUMMARY
This analysis is intended to aid Region 10' s Radon and Indoor Air Program
managers in their efforts to ...
information rather than targeted campaigns. EPA might gain cooperation by making
its staff more available.
Recommendation ...
II. Radon and Indoor Air: National Program Design
The Indoor Air program was initiated by the Radon Gas and Indoor Air Qua...
r
It is important at the outset to settle on a definition of outreach. An
expansive definition may be best since almost ev...
• Most radon testing (70-90%) occurred in the context of real estate
transactions.
• State and local public information an...
III. The Bennett Model
A blueprint for planning and measuring outreach activities was developed by
Claude Bennett of the D...
Bennett's Model*
Results
(which are reached by)
Behavior Changes
(which result from)
KASA (Knowledge - Attitude - Skill - ...
IV. Explanation of Bennett's Hierarchical Steps
Results: This step asks the essential question, how do you know when
you h...
What is essential at this stage of program design is that a clear, readable,
technically accurate body of information be c...
Bennett recognized the limits of quantitative measurement -- the difficulties of
measurement and the substantial resource ...
V. The Bennett Model Applied in Washington
Two agencies of Washington state government have applied the Bennett Model in
d...
The Puget Sound Water Quality Authority (PSWQA), originally established by the
Legislature as an independent agency and no...
• Environmental outreach may be difficult to measure during the life of a
proj ect. Proj ects tend to reflect funding cycl...
• Program design, on the macro level, may inhibit the accomplishment of
goals. For example, one of the Radon program's pri...
VI. Outreach in Other Government Agencies
Most of the EPA regional offices, along with a few other government agencies,
we...
VII. Outreach in Region 10
Region 10 performs an impressive amount of outreach in the Indoor Air and Radon
programs. Staff...
VIII. RECOMMENDATIONS
Recommendation 1: Maintain a base program of general outreach designed to
increase contacts with the...
But very little of their activity shows any results. True, they did have some
involvement in the passage of legislation th...
Recommendation 4: EPA Region 10 can developing strategic plans with its grantees
and/or the states.
While federal, state, ...
Appendix 1
Puget Sound Water Quality Authority
PIE Fund Questionnaire
Pertinent portions of the post-project questionnaire...
6.3 When did these activities occur?
6.4 What externalfactors affected the success of these activities? (bad weather, no
c...
r~'
I
Appendix 2
Survey of Outreach in Other Government Agencies
Agency Name Staff Outreach Methods Employed Effectiveness...
EPA Reg. 3 Presentations to comm'ity grps; Setting up tables at malls not very effective; Don't measure effectiveness, but...
Appendix 3
WSDOH State Indoor Radon Grant Program
Final Report: July 1, 1991 - June 30, 1992
• Report period represent Yea...
Upcoming SlideShare
Loading in …5
×

Analysisof outreachinthe radonindoor air prog0001 1

396 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
396
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Analysisof outreachinthe radonindoor air prog0001 1

  1. 1. ANALYSIS OF OUTREACH IN THE RADON AND INDOOR AIR PROGRAMS Region 10 Policy, Planning and EvaJuation Branch May, 1993
  2. 2. Background Region 10's Radon and Indoor Air Program managers requested the Policy, Planning and Evaluation (PP&E) Branch to conduct an analysis of the effectiveness of their outreach activities. The main goal was for PP&E to help the Radon and Indoor Air Program (RIAP) determine how they can optimize efforts to direct their messages to specific target audiences. Analysis success is the development of recommendations and information that allows RIAP to measure and demonstrate that they are directing their limited resources in ways that maximize the reduction of risk. The RIAP wanted recommendations that would aid efforts to target their outreach toward geographic areas, audiences, and pollutants with the greatest risk. Overall Program guidance is established by HQ, but the Region has a wide range of methods available for disseminating and implementing the guidance. RIAP has limited resources. The PP&E Branch was to develop recommendations that can be implemented without foreseeable increases in resources, .and recommendations for RIAS time allocation based on quantitative measures of program impact. RIAS requested preliminary results available in time to incorporate into their planning for FY94. 2
  3. 3. ••• TABLE OF CONTENTS I. Executive Summary Page 4 II. Radon and Indoor Air Pollution 5 III. Bennett Model 8 IV. Explanation of Bennett's Hierarchical Steps 10 V. Bennet Model Applied in Washington 13 VI. Outreach in Other Government Agencies 17 VII. Outreach in Region 10 18 VIII. Recommendations 19 Appendices: Appendix 1 22 Appendix 2 24 Appendix 3 26
  4. 4. EXECUTIVE SUMMARY This analysis is intended to aid Region 10' s Radon and Indoor Air Program managers in their efforts to determine the effectiveness of their outreach activities. The Radon and Indoor Air Program (RIAP) strives to optimize their efforts to successfully target audiences and deliver messages to protect the public. The analysis presents recommendations and information that will help RIAP to measure and demonstrate that they are directing their limited resources in ways that maximize the reduction of risk and public protect. The dichotomy between outreach that is "shotgunned" and that which is more precisely targeted need not necessarily be resolved entirely in favor of one or the other. This analysis will not offer recommendations that support or reject either approach. A program that includes both approaches to outreach is much more realistic. These recommendations are intended to suggest different approaches that are not exclusive of one another. The follow"ing recommendations are offered to address the analysis goals (all recommendations are discussed in more detail in the Recommendations section): 1. Maintain a base program of general public information. 2. Build feedback/measurement systems into our grants and contracts. 3. Begin conducting our own feedback/measurement activities. 4. Develop strategic plans with our grantees and/or the states. 5. Develop an outreach capability within the Office of External Affairs. Recommendations Recommendation 1: Maintain a base program of general outreach designed to increase contacts with the general public and with key participants. Radon/Indoor Air Quality programs are in their infancy. A general dissemination of information will help to develop an infrastructure of informed people who can participate in a slow but steady shift of opinions and behavior. This approach is consistent with the patterns followed by our grantees and contractors -- mainly the states -- who have the power to reach more people than we do. Too narrow of an approach reduces the number of potential allies who may at some point, through their purchasing habits, votes, word-of-mouth, or other behavior, contribute to better indoor air quality. Recommendation 2: Build feedback/measurement systems into EPA grants and contracts. Disbursed funds are a primary lever to achieve better indoor air quality. This funding is an excellent opportunity for determining how well our outreach is working within the public sector. This Recommendation advocates building limited feedback and measurement into grants. Instead of devoting the entire grant to general public information, the grantee can craft a proj ect of limited scope that lends itself to design and monitoring. This approach requires gai.m.ng t;heapproval and cooperation of the grantee, which might be a challenge given their limited resources and preference for general public 4
  5. 5. information rather than targeted campaigns. EPA might gain cooperation by making its staff more available. Recommendation 3: activities. EPA Region 10 can conduct its own feedback/ measurement To date there have been many public presentations, training sessions, informational inquiries, distribution of materials, and various other people who might be characterized as members of a radon/indoor air "infrastructure." We could survey them to find out how useful our information was and how it led to changes in behavior. More specifically, we could poll real estate brokers to develop a data base of tested homes. We could also query radon testing and mitigation contractors to determine their involvement in reducing radon risk. Recommendation 4: EPA Region 10 can developing strategic plans with its grantees and/or the states. While federal, state, and local government may still be struggling through the start-up phase of radon and indoor air management, we can look ahead. This is a uniquely federal role, given our four-state perspective, the nation-wide effort at the national level, and the leverage we possess as grantor. Programs and projects that lend themselves to feedback and measurement may stretch over longer periods than the typical one-year grant. We need to persuade others to work with us to develop a sense of what priorities are needed within our programs and how we can achieve measurable results. This proposal is not limited to the states; there are non-profit organizations, such as the Washington Lung Association, who are potential collaborators. We can consider forming alliances, in the form of cooperative agreements, with major collaborators. This encourages a common solution, as well as allows leveraged resources. Cooperative methods, driven by strategic planning, allow for feedback and measurement of everyones' effectiveness. Recommendation 5: Affairs. Develop an outreach capability within the Office of External Neither Radon nor Indoor Air have had much involvement with the Office of External Affairs. There has been some press interest, but the interest fluctuates. Press is, by definition, concerned with the news value of an occurrence. Outreach is a continuing, rather than an episodic, activity. Outreach can be augmented by professional capabilities (writing, graphics, education) which the Office of External Affairs could provide as needed by Radon, Indoor Air, and other programs. The Policy, Planning and Evaluation Branch believes this analysis can be useful to the Radon and Indoor Air Programs. Further analysis and/or support is available should the program desire it. 5
  6. 6. II. Radon and Indoor Air: National Program Design The Indoor Air program was initiated by the Radon Gas and Indoor Air Quality Act of 1986, Title IV of the Superfund Amendment and Reauthorization Act (SARA). The legislation established a research program to identify, characterize, and monitor the sources and levels of indoor air pollution, investigate the human health risks, and to develop control technologies or other mitigation measures. The statute also provided for disseminating information to the public about the research activities. EPA has expanded on the statutory mandate through its FY 93 program guidance, which emphasizes outreach activities at the Headquarters level: research, preparation of guidance documents for physicians and architects, documents for targeted segments of the public, an information clearinghouse, written materials on assessment and mitigation with private sector organizations, and training. The regional office role includes public information, technical assistance, training, and state and local program development. The Indoor Radon Abatement Act, passed in 1988, relies on similar measures. It provides for the publication of a citizen's guide that outlines the health risk from radon, mitigation costs and techniques, and testing methods. EPA is further directed to develop model construction standards and provide technical assistance to state radon programs, including training, an information clearinghouse, demonstration programs, and publication of public information materials concerning radon health risks and methods of radon mitigation. A key aspect of the radon program, at least from the perspective of EPA regional offices, is state grant funding which is authorized to states for building and geographic surveys, and purchase of measurement and analytic equipment. Other potential activities include development of public information and educational materials, developing data systems, mitigation demonstration programs, and toll-free radon hotlines to provide information and technical assistance. EPA Headquarters guidance for FY 93 identifies five elements to be addressed by the radon program: 1. Problem Assessment (identify areas with high radon levels) 2. Mitigation and Prevention (methods to reduce or prevent radon contamination) 3. Capability Development (improve state, tribal, private sector, and general public in dealing with radon) 4. Public Information (provide timely information about radon risk and methods of risk reduction) 5. Federal Coordination (work with other federal agencies and buildings) While public information is presented as a separate prong of this five-part strategy, the last three of these elements arguably constitute outreach activities. The guidance also iden-tifies regional office roles: overseeing regional training centers, promoting new building codes; overseeing state grants; promoting real estate guides and addressing radon problems during real estate transactions; representing EPA to state, local, tribal, and private sector; providing technical support; and expanding public outreach activities. This last role includes developing public education materials (public service announcements, brochures, posters), providing EPA publications to states and tribes, and promoting model building codes to state and local governments. Again, public outreach is an enumerated function, but all of the listed regional office roles probably constitute outreach. 6
  7. 7. r It is important at the outset to settle on a definition of outreach. An expansive definition may be best since almost every activity in the radon and indoor air programs constitutes outreach. The term encompasses more than just public information activities (e.g. production and distribution of written information, public service announcements, press releases). Training, grants, promotion of changes in building codes or real estate agent licensing requirements -- all are opportunities to educate and persuade, which I believe is what we intend when we employ the word "outreach". The national program structures for radon and indoor air invite some conclusions: • They are voluntary programs that rely on research and outreach rather than regulatory strictures. • They reflect a national (some would say Beltway) approach that emphasizes broad, sweeping measures. to policy-making • They provide no guidance on the precise nature of the outreach to be conducted by the regional offices. They offer none of the precision and narrow focus that is conducive to gathering useful feedback and measuring program effectiveness. On the other hand, this leaves the regional offices with enormous latitude in their outreach practices. • Both statutes and programs are first-time, start-up efforts that forsake the regulatory approach in favor of behavior change by promoting "a better way" through education and persuasion. EPA has further fleshed out its program goals through a radon program analysis to determine how well the program has been working. Draft Radon Program Review, Office of Policy, Planning and Evaluation, Environmental Protection Agency, May 1992. The study found that EPA's radon program had raised public awareness, made progress in testing and mitigating, supported state programs, and collected data on radon distribution and health effects. However, the study found that only limited testing and mitigation had occurred; about six percent of homes had been tested and even fewer of those needing mitigation had been mitigated. The analysis further found that, although early program success compared favorably with similar efforts in anti-smoking, drunk driving, seat belt and crime safety, EPA wanted to have a greater impact on radon health risks. The study made four major recommendations: 1. EPA should focus on the greatest risks first (high radon potential areas and smoking). 2. EPA should emphasize the long-term strategies of promoting radon-resistant new construction and testing/mitigation in connection with real estate transfers. 3. EPA should develop a coordinated research plan. 4. EPA should move beyond a nation-wide public information program and develop a new strategy for public information, one that emphasizes high radon potential areas and that focuses on building support for construction of radon-resistant new homes and for testing/mitigation of existing homes when they are sold. The fourth recommendation, regarding public information, directly concerns us here. The study made several findings about public information, including: 7
  8. 8. • Most radon testing (70-90%) occurred in the context of real estate transactions. • State and local public information and incentive campaigns have yielded limited responses. A public information effort in Washington, D.C., resulted in a one-time testing rate of about four percent. Local direct mail and free or discount testing kit distribution have resulted in one-time testing rates of 15- 30%. • Government has a primary role of informing the public, particularly because radon is imperceptible without a public information effort. • A foundation of public awareness is necessary to reduce radon risk. • EPA's public information campaign has raised public awareness; 80% of the population has heard of radon and 65-70% understand that it poses a health risk. • Nevertheless, public information alone is not sufficient to make a significant impact on radon risk reduction; EPA should be moving beyond a nation-wide public information strategy. • The localized nature of radon limits the federal role; states but federal energies can complement through research, pilot and projects, and support for state activities. Other health organizations, at all levels, also have roles. have the lead demonstration and consumer The Headquarters analysis sends mixed signals about outreach. On the one hand, the report finds a need to move beyond a broad-brush public information program and identifies a need to narrow the focus of outreach and recommends where it can be done -- real estate transactions. This attempt to isolate an aspect of the overall radon problem seems consistent with the Bennett Model (see below), which works best when the results sought are narrowly drawn. 8
  9. 9. III. The Bennett Model A blueprint for planning and measuring outreach activities was developed by Claude Bennett of the Department of Agriculture in 1974. Bennett was trying to analyze the effectiveness of the numerous outreach programs administered through Cooperative Extension. Washington State government has found the model useful; both the Department of Wildlife and the Puget Sound Water Quality Authority have applied it to their programs. In creating the model, Bennett wanted to provide a useful tool for program managers to know whether their program was succeeding. Bennett's model assumes that program evaluation (what we are calling measurement of the effectiveness of outreach) is built into every program from its outset, instead of juryrigging a measurement system sometime down the road to provide hindsight. The model consists of seven hierarchical steps. Each step represents a category of outcomes which, in turn, allows us to develop criteria for evaluating the outcomes. The seven steps are arranged along a continuum, with "ends" at the higher end and "means" at the lower end. Each step is a prerequisite to the next, in terms of program design. However, actual program planning is conducted in reverse -- you establish your desired results and work backward toward inputs. The essence of the model is that you cannot design a program unless you know in advance, with specificity, what your program is supposed to accomplish. Bennett's Model is a simple one; it organizes obvious program components in a logical progression in a way that encourages a program planner or manager to see the connections and integrate them in a result-oriented fashion. [Bennett's Model is illustrated on the next page.] 9 I
  10. 10. Bennett's Model* Results (which are reached by) Behavior Changes (which result from) KASA (Knowledge - Attitude - Skill - Awareness (which are influenced by) Reactions (which are a function of) Participants (which happens due to) Activities (which are offered as a result of) Inputs * Designing an Effective Communication Program: A Blueprint for Success; University of Michigan: School of Natural Resources and Environment; Prepared for U.S. Environmental Protection Agency, Region 5; September 1992; page 60. ' 10
  11. 11. IV. Explanation of Bennett's Hierarchical Steps Results: This step asks the essential question, how do you know when you have done your job? Someone must identify a problem and then enunciate a goal so that we know when the problem is solved. An important question to be resolved is, who determines the goal (s) -- policymakers or those charged with conduct of the outreach effort? There are three kinds of results: 1. Environmental -- a change in the condition of the environment (e.g., a renewed fish run; a measurable improvement in opacity; a ban on lead in gasoline). 2. Protective mechanism in place -- compliance with a measure that is deemed to yield some benefit (e.g., obtaining a Clean Water Act §404 permit; implementation of the National Contingency Plan for Superfund sites). 3. Institutional capacity -- this is a pseudo-result, where a regulation is approved or a program is fully funded (e.g. listing a Superfund site on the National Priority List) When program design is initiated at the "Results" stage, the goal can be expressed in terms of qualitative or quantitative measures. Much of EPA's outreach appears to have considered this step in only the most general way -- if we disseminate information widely then people will smarten up and we will have a better environment. A glance through the Bennett model reveals how this method has skipped many steps that provide the logical connections between program and results. EPA outreach resembles the reproductive methods of many plant and animal species: scatter enough seeds and some will take root. While this may work for Douglas fir, it may not reflect the most careful husbandry of public resources, nor does it protect public health in the most comprehensive or efficient manner. Behavior Change: This involves identification of some durable, persistent institutional or personal behavior change (e.g., stop smoking; buy recycled materials). Program designers should ask, what are the cultural enforcers or inhibitors of behaviors? Behavior change can be difficult to measure it is impossible to know how many have quit smoking. Then we need to look for analogue or indicator behavior -- cigarette sales, expansion of anti-smoking legislation. Follow-up may be necessary to determine behavior changes, such as telephone surveys. KASA (Knowledge - Attitude - Skill - Awareness) This is the information component of program design, where we consider what people think and know and what they should be thinking and knowing in order to alter behavior and achieve program results. Ideally, we conduct a pre- test of KASA to determine what people knew prior to program implementation. A post-test after the program should then tell us the effect of the program. These are very resource and time-intensive measures. 11
  12. 12. What is essential at this stage of program design is that a clear, readable, technically accurate body of information be compiled. It may have to encompass a range of values in order to give everyone targeted what they need to change behavior. Often, program design fails to go past this step in determining program effectiveness Behavior Change and Results are not considered. Unfortunately, without feedback about what the outreach has accomplished, we are just "bean-counting" (e.g., we published X brochures and we reached a potential audience of Y through radio public service announcements). Reactions For outreach efforts to work, we have to create a climate of learning: timing, surroundings, consider the feelings of participants, appropriate behavior by "teachers." Everything must be structured to achieve "buy-in" by participants. Participants The outreach audience must be targeted with prec1s1on. The larger and more diverse the audience, the less likely a message can be tailored that achieves the behavior changes that will produce the result wanted. If the audience is the general public, we will likely be reduced to a general "shotgunning" of information accompanied by a leap of faith that appears to be based on the theory that we know the facts and we understand the health risks, and, therefore, if we publicize what we know, everyone will know this and act wisely to protect their health. There is no evidence that broadbrush outreach accomplishes this. For an identified audience, we want to know what "pushes their buttons", how they can best be approached. Perhaps affinity groups (labor unions, churches, professional societies) can be used as conduits to reach an audience. Activities This step details the tools or levers available to augment a program: workshops, conferences, technical assistance, toll-free hotlines, brochures, speeches, testimony, etc. Activities encompasses more than the capabilities of our own agency; can we leverage additional performance through grants, contracts, cooperative agreements? These activities can be arranged into a timeline to organize the entire program and envision what will occur. Enputis ; The resources to be applied to the program include FTE, extramural funding, volunteers, etc. In a program planning sense, this step of the hierarchy requires budgeting for results. Having designed the program, committing to outcomes, and identifying the resources will eventually lead to those results. 12
  13. 13. Bennett recognized the limits of quantitative measurement -- the difficulties of measurement and the substantial resource implications. In response, he also developed a qualitative measurement system for program evaluation. He called this Reflective Appraisal of Programs (RAP); program participants "reflect" upon the changes effected by a program. In essence, participants are asked to respond to a standard questionnaire keyed to any of the four "higher end" outcomes of Bennett's hierarchy -- Reactions; K.A.S.A. Change; Behavior Change; and Results. The RAP approach encourages use of the findings from the interviews to draw conclusions about and evaluate results of the program. These conclusions then lead to recommendations about how decision-makers can apply the findings. The Bennett Model offers a systematic, logical, easily understood framework for program design. Its emphasis on backwards planning, with project results serving as the key to project design, encourages a focus on project feedback or measurement. 13
  14. 14. V. The Bennett Model Applied in Washington Two agencies of Washington state government have applied the Bennett Model in different ways. The Department of Wildlife conducted a retrospective analysis of its Project WILD, an environmental education effort that had been in operation for eight years. The goal of Project WILD was to "...assist learners of any age in developing awareness, knowledge, skills, and .commitment to result in informed decisions, responsible behavior, and constructive actions concerning wildlife and the environment upon which all life depends." Evaluation of Project WILD: The State of Washington 1984 - 1992, by Margaret Tudor, Washington Department of Wildlife, 1992. When measured against the Bennett Model, Project WILD was found to have many of the preliminary prerequisites for program success. For example: Inputs: 1 or 2 FTE throughout program life; 1,000 elementary guides, 300 secondary guides, 300 aquatic guides produced annually. Activities: 376 introductory and advanced workshops held; 2 conferences held (total attendance 800). Participation: 6,300 teachers (15.5% of total) participated. Beyond this point in Bennett's Hierarchy, program evaluation of Project W+LD tended to be more anecdotal: Reactions: Survey found response to workshops favorable; teachers reported curricular resources useful. K.A.S.A.: Teachers report change in students' attitude, but students', knowledge, attitude and skill change not measured. Behavior change: No strong evidence directly linking Project WILD with wildlife protection and habitat enhancement; anecdotal evidence that advanced workshops using Project WILD as resource has led to habitat enhancement by .teachers. Results: Project WILD -- no effects measured; anecdotal evidence that habitat enhancement education program linked to habitat grants protected or enhanced 3,000 acres. The analysis concluded that Project WILD needed further development of its goals, and that the goals had to be expressed in terms of measurable outcomes. Targets to meet the goals needed to be set at the outset of the program. 14
  15. 15. The Puget Sound Water Quality Authority (PSWQA), originally established by the Legislature as an independent agency and now a part of the Department of Ecology, has conducted an ambitious outreach program for several years. Through the Public Involvement and Education (PIE) Fund, PSWQA has made an experimental effort to involve the public in managing Puget Sound through a number of creative outreach activities. PSWQA conducted a study of 42 PIE Fund projects, measured in terms of Bennett's hierarchy. Publicly Financed Voluntary Initiatives for Public Involvement and Education as a Natural Resource Management Tool, An Evaluation 1989-1991 Washington State Public Involvement and Education Projects, Funded by the Puget Sound Water Quality Authority, July 1991. The study include the following findings: • All projects (100%) fulfilled Levels 1,2 and 3 (Inputs; Activities; Participation) in Bennett's model. • 90% of the projects were fulfilled through Level 4 (Reactions). • At Level 5 (K.A.S.A. changes) 71% were fulfilled. • 33% reached Level 6 (Behavior/Practice changes). • 14% reached Level 7 (Results) -- some form of water quality protection or improvement. The PSWQA and Department of Wildlife studies cry out for additional analysis -- so here goes. The lower end, early stage steps in Bennett's hierarchy are more readily quantifiable. At that point in program planning we are measuring resource levels, numbers of people, workshops, conferences, and hotline calls answered. As we move up the scale away from "means" toward "ends", our methods of measurement are few -- we can conduct surveys, send out questionnaires. These measurements tend to give us qualitative, subjective responses which usually tell us that we are having some effect; but we have no way of knowing whether we have orchestrated an efficient use of public resources. Low return rates on these measurements leave us questioning whether we have a representative sample or whether we really even know what effect our program has had. It is even more difficult to determine what behavior changes we have effected; notice how the results in the PSWQA study drop from 71% (K.A.S.A. changes) to 33% for Behavior changes. The Project WILD study, while not as quantitatively precise, shows a dramatic change in the nature of the outcomes at the higher end of Bennett's scale. In both studies, the evaluations of Results suggest that the projects ran out of gas; Project WILD had inadequate anecdotal information, and PSWQA found that only 14% of the projects fulfilled their purpose. Some possible explanations for this dropoff in measurable performance are: • Program design has not included more precise measurement. • Environmental and natural resource outreach proj ects are inherently difficult to measure if they focus on the effects on the natural environment. Puget Sound or the ambient atmosphere are too big and there are too many potential intervening factors that may mask our projects, however well-intentioned, -designed, and -implemented they may be. 15
  16. 16. • Environmental outreach may be difficult to measure during the life of a proj ect. Proj ects tend to reflect funding cycles and budget appropriations. Perhaps measurable effects are longer term phenomena that do not readily fall within these timeframes. • Perhaps it is too early to tell. Many government outreach programs are still in their infancy. Managers and staff may not have been trained in the importance of outreach or in its intricacies. Development of outreach skills may elevate the outcome of future projects.Measurable results may not be available until some time in the future. • Measurement at the higher end of Bennett's hierarchy requires substantial resources. It may involve significant socialfbehavioral science field work. At the KASA level this involves the art of learning assessment. Project evaluation of this magnitude is normally beyond the means (funding and time) of public agencies. • The lack of quantifiable data at the higher end of the model may simply reflect a bias of environmental/natural resource agencies: our natural bent is to look for things to measure. When we cannot find them, we regard whatever feedback we get as suspect or inadequate. Maybe we need to reassess our own attitudes and accept qualitative evaluations. • There may be a form of entropy in human affairs that applies the longer an undertaking lasts, or the greater the span of control among the individuals involved. In government agencies, resource cuts, personnel turnover, and shifting priorities may take the bloom off the rose before a proj ect is completed. Outreach depends on influencing people and activities that are not within the control of a government agency and which have other demands on their time, enthusiasm, and loyalty. Perhaps there is simply a natural enthusiasm at the outset of a project that is inevitably drained <}:waybefore completion. Perhaps measuring program success requires more, or different energies and attitudes, than setting a project in motion does. 16
  17. 17. • Program design, on the macro level, may inhibit the accomplishment of goals. For example, one of the Radon program's primary tools is grant funding to the states. However, federal law requires a 50% state match for federal funding. Scarce radon funding on the state level has resulted in less than 100% of available federal grant money being awarded in Region 10. If we are not using all of the admittedly low level of program resources, it should not surprise us that our outcomes fall short. PSWQA has taken steps to overcome the difficulties of measuring outreach effectiveness. The Authority has designed a questionnaire for use by PIE Fund grant recipients to analyze the effectiveness of a grant. (See Appendix 1) PSWQA uses a pre-proj ect and post-proj ect questionnaire for grant recipients. The two questionnaires differ essentially only in verb tense. They are closely keyed to the Bennett Model. The questionnaires are designed to have grantees thinking in terms of results at the inception of the project, and to reinforce the importance of thorough program design as the project is completed. 17
  18. 18. VI. Outreach in Other Government Agencies Most of the EPA regional offices, along with a few other government agencies, were informally surveyed by telephone to determine what outreach techniques they were using arid their success in measuring outreach effectiveness. (See Appendix 2) As far as outreach methods, nothing novel emerged -- they were using anticipated tools to predictable audiences. Almost without exception they claimed no success in measuring outreach. The Great Lakes Program (EPA, Region 5) relies on a carefully designed survey of visitors to a ship that sails among Great Lakes ports to ascertain visitors' knowledge of water quality issues. They are surveying a captive audience for information purposes only; no behavior change is sought. There is little applicability of that effort to radon and indoor air. A few of the other offices surveyed count basic information such as the number of people reached by various outreach methods, but none has found a way to measure changes in behavior. Those offices polled expressed an interest in this study as a means of finding better ways to chart their effectiveness. 18
  19. 19. VII. Outreach in Region 10 Region 10 performs an impressive amount of outreach in the Indoor Air and Radon programs. Staff are fielding public inquiries, developing contacts in public and private sectors, organizing and presenting training, meeting with key groups, circulating written materials, appearing at horne shows, etc. There have been attempts to target audiences, such as appearances before real estate brokers' associations regarding radon, which is consistent with the recommendation in the Headquarters Radon Program Review that real estate transactions be a primary focus of outreach. Staff are maintaining mailing lists and identifying attendees at various events. There has been no lack of energy in trying to promote both programs. What is less apparent is whether the programs are using a measured, result- oriented approach in their outreach -- the Bennett Model or something similar. Their efforts seem more concerned with reaching as many different audiences in as many different ways as possible, rather than targeting an audience with a particular result in mind and then trying to measure the result. For example, the Indoor Air Coordinator participated in the presentation of the training program "Orientation to Indoor Air Quality" in Spokane. If this particular outreach event is analyzed in light of the Bennett Model, we discover the following: Inputs: The Region's lone program coordinator (1 FTE) disseminating program knowledge. Activities: Three-day interactive training program. Participants: Federal, state, and local employees, representing a variety of agencies, non-profit organizations, and some private sector representatives, many from Washington and Idaho. The implicit purpose of this training is to impart information, raise consciousness, develop networks of information and allies, and generally bring key individuals to a higher level of understanding of indoor air issues. However, this effort did not include any attempt to determine the Reactions of the participants, the Knowledge - Ability - Skills - Attitudes of the participants before and after the training, their Behavior Changes, and the program Results of the training. An attempt to measure the effects of the training would have included these steps. (No criticism of any individual is intended by this analysis; it is offered solely to suggest program improvement, or at least to illustrate a different approach.) The traditional broadbrush approach to outreach and the more focused approach offered by the Bennett Model highlight the fundamental tensions in performing outreach, which is the main point of the Recommendations offered here. 19
  20. 20. VIII. RECOMMENDATIONS Recommendation 1: Maintain a base program of general outreach designed to increase contacts with the general public and with key participants. Government has an educational function. Increasing the number of educated potential allies who may at some point, through their purchasing habits, votes, or other behavior, contribute to better indoor air quality, is an educational role. Radon/Indoor Air Quality programs are in their infancy. A general dissemination of information to develop an infrastructure of informed people who can participate in a slow but steady shift of opinions and behavior. This approach is consistent with the patterns followed by our grantees and contractors -- mainly the states -- who have the power to reach more people than we do. We can explore ways to make this general information function more efficient than it is. Perhaps we can make morejbetter use of Senior Executive Employees (SEE) by giving them primary responsibility for responding to general telephone inquiries. This frees the program coordinators to perform new duties suggested within these Recommendations. An alternative that could produce a similar result would be to use the Public Information Center in the Office of External Affairs for answering public inquiries. This would require giving PIC staff more training and more authority in answering questions, instead of only furnishing the public with publications. There are other difficulties: these are contractor personnel and we would have to decide whether contractors should be performing this function. On the other hand, this might be a preliminary step in creating a more omnibus public information function in OEA, which would broaden that Office's involvement in Regional affairs and free program staff for other, more narrowly focused outreach activities. Recommendation 2: contracts. Build feedback/measurement systems into EPA grants and Disbursed funds are a primary lever to achieve better indoor air quality. This funding is an excellent opportunity for determining how well our outreach is working within the public sector. Appendix 3 is a summary of the Washington State Department of Health's ac tivi ties under an~PA grant in the Radon Program for a one-year period. Staff made a host of presentations, attended meetings, drafted and updated fact sheets, operated a toll-free information line, issued press releases, and published articles. 20 I
  21. 21. But very little of their activity shows any results. True, they did have some involvement in the passage of legislation that requires local building officials to provide radon testing devices in new home construction. That is a result of sorts; of course, a more meaningful result would describe the mitigation that occurred as a consequence of testing. Other activities suggest tantalizing possibilities of feedback/measurement. They conducted a testing survey and mailing survey (of what? to what end?). They backed away from a plan to test and mitigate economically disadvantaged homes when the contractor failed to produce matching funds. Instead a general public information effort was introduced. This Recommendation advocates building limited feedback/measurement into grants. Instead of devoting the entire grant to general public information, the grantee can craft a project of limited scope that lends itself to design and monitoring as envisioned by the Bennett Model. For example, isolation of real estate transactions within a given area, involving local realtors, offering test kits or services at a reduced rate, conducting public information campaigns, and then measuring tests at the time of real estate transactions can create feedback. This approach requires gaining the approval and cooperation of the grantee, which might be a challenge given their limited resources and preference for general public information rather than targeted campaigns. EPA might gain cooperation by making its staff more available (with the time available through the use of Office of External Affairs (OEA) personnel to field- inquiries -- see #l above). Recommendation 3: activities. EPA Region 10 can conduct its own feedback/ measurement To date there have been enough public presentations, training sessions, informational inquiries, distribution of materials, and various other people who might be characterized as members of a radon/indoor air "infrastructure." We could survey them to find out how useful our information was and how it led to changes in behavior. More specifically, we could poll real estate brokers to develop a data base of tested homes. We could also query radon testing and mitigation contractors to determine their involvement in reducing radon risk. Again, the available staff time through public inquiries fielded by individuals other than program staff could allow focus on this activity. 21
  22. 22. Recommendation 4: EPA Region 10 can developing strategic plans with its grantees and/or the states. While federal, state, and local government may still be struggling through the start-up phase of radon and indoor air management, we can look ahead. This is a uniquely federal role, given our four-state perspective, the nation-wide effort at the national level, and the leverage we possess as grantor. Programs and projects that lend themselves to feedback and measurement may stretch over longer periods than the typical one-year grant. If we are serious about gathering this information, we need to persuade others to work with us to develop a sense of what priorities are needed within our programs and how we can achieve measurable results. This proposal is not limited to the states; there are non-profit organizations, such as the Washington Lung Association, who are potential co~aborators. We can consider forming alliances, in the form of cooperative agreements, with major collaborators. This encourages a common solution, as well as allows leveraged resources. Cooperative methods, driven by strategic planning, allow for feedback and measurement of everyones' effectiveness. Some of our grantees have continued with us for two or more consecutive years. The Alaska Health Project (a non-profit) has been a grantee for FY 91-93, as has the Washington State Department of Health. The Oregon Department of Environmental Quality has been a grantee for FY 92-93. These patterns suggest some commitment, assuming continued grant funding. There are potential pitfalls -- grantees are taking the risk that EPA funding will continue, they will be chosen as grant recipients, and program priorities will remain constant enough for multi-year undertakings seen through to conclusions. Recommendation 5: External Affairs. Develop an outreach capability within the Office of Neither Radon nor Indoor Air have had much involvement with the Office of External Affairs. There has been some press interest, but not recently. Press is, by definition, concerned with the news value of an occurrence. Outreach is a continuing, rather than an episodic, activity. Outreach can be augmented by professional capabilities (writing, graphics, education) and could be located in OEA for use as needed by Radon, Indoor Air, and other programs. 22
  23. 23. Appendix 1 Puget Sound Water Quality Authority PIE Fund Questionnaire Pertinent portions of the post-project questionnaire are listed below. Section 1: Improving Water Quality in Puget Sound 1.1 What condition has your project helped correct? 1.2 Has that condition been corrected? 1.3 How do you know the condition has been corrected? 1.4 How has your project contributed to correcting the condition? Section 2: Individual and Group Behavior 2.1 What behaviors (or lack of) on the part of individuals or groups contributed to the condition described in Section 1? 2.2 What changes in behavior do you hope have occurred as a result of your project? 2.3 How do you know if the behavior of project participants has changed as a result of your project? 2.4 Do you plan any followup activities to estimate long-term changes in behavior of individuals or groups involved in your project? Section 3: Knowledge, Attitudes, Skills, or Awareness 3.1 What information did project participants receive as a result of your project? 3.2 How was that information transmitted to them? 3.3 How did you find out if they received, understood, or used the knowledge, attitudes, skills or awareness that they received during your project? Section 4: Reactions Expected from Project Participants 4.1 Under what situations did learning occur in your project? 4.2 How do you hope project participants felt about your project or Puget Sound after being involved in your program? 4.3 How do you know how they feel? 4.4 Were project teachers (or others presenting intormatlon, if applicable) evaluated by participants or audience? If so, how? Section 5: Project Participants 5.1 Who was the audience for your project (be as specific as possible)? 5.2 What was the most effective way you found to publicize your project? 5.3 How many direct participants (workshop attendees, viewers, listeners, publication readers, site visitors, etc.) were involved in your project? 5.4 How many indirect participants were involved in your project? 5.5 What products (publications, etc.) did your project produce? How many? 5.6 Who participated in technical review or program oversight? Section 6: Project Activities 6.1 What activities or events were involved in your projects? 6.2 What did participants do at activities or events? 23
  24. 24. 6.3 When did these activities occur? 6.4 What externalfactors affected the success of these activities? (bad weather, no competing events, etc., cooperation of another party or organization)? Section 7: Resources Put to Work 7.1 What was the total budget for your project? 7.2 How much money was raised from sources other than PSWQA? 7.3 What other kinds of contributions were made to the project? 7.4 How many paid employees worked on the project? For what percentage of their workload? 7.5 How many volunteers worked on the project? Section 8: Problems and Challenges What problems or challenges were faced in completing the project and were they dealt with? Section 9: Unanticipated Results 9.1 Describe unanticipated successes (other than planned goals and objectives) that you achieved in completing this project. 9.2 To what do you attribute these? 24
  25. 25. r~' I Appendix 2 Survey of Outreach in Other Government Agencies Agency Name Staff Outreach Methods Employed Effectiveness How Measured Army Corps of Eng'rs Press News releases Good questionl Counting beans (# of news Count # of papers that pick up Ofc. releases, speeches, or public inquiries) not news releases or inquiries that accurate result from news release; hiring P.R. firm to research O/R very expensive Seattle City Light Com Newsletter; public presentations Trade shows and community events allo~ Customer Service Survey shows munit them to meet their customers how many customers they inform; y use signup sheets at presentations Relati to count # reached; don't know ons (5 how to measure whether public is staff) "getting" their Info WA Dept of Ecology Need to plan measurement upfront, In program design; easy to measure # of contacts, harder to measure changes in people's behavior; try to measure changes in attitudes and awareness? EPA Reg. 1 Presentations for groups; booths at Most effective when they target specific Count # of cases; use signup fairs; speeches; written groups sheets at presentations - send publications; show test kit questionnaires; track calls - correlate w/groups they have given presentation; clip articles from newspapers that use their info EPA Reg. 2 Respond to calls; offer tech asst; Want to use Am. Lung Assoc. to followup IAQ: program too new to tell send written materials; bldg. AQ course to avoid "survey" issue; problem w/O/R is that it leads to more calls if effective, requiring more staff; no single form of O/R works -- need combo 25
  26. 26. EPA Reg. 3 Presentations to comm'ity grps; Setting up tables at malls not very effective; Don't measure effectiveness, but developed school curriculum - haven't identified audience have good data re students take home test kits and testing/mitigation - see direct must return; hot line for public correlation between D/R and inquiries; full color maps for testing/mitig distrib.; use questionnaire for speakers: who/how many attended presentations; lots of radio PSA's; work w/Better Breather's Clubs; health fairs at hospitals and firms; work w/states, realtors; combine w /EED program in work w/schools; coord w/ALA re Rn Awareness Mo. EPA Reg. 4 Provide speakers; presentations for Most effective when targeting specific grps. Have no way of measuring gps. EPA Reg. 5 Use sophisticated survey Survey measures what people techniques re Great Lakes project learned from ship - nothing specific re Rn/IAQ EPA Reg. 6 1 FTE Presentations to schools Unknown Don't measure EPA Reg. 7 Brochures, talks in schools, PSA's D/R aimed at children seems effective, based Don't have resources to measure on experience re smoking and seatbelts; PSA's waste money - played in dead time EPA Reg. 8 _ IAQ program just trng state staff; Too early to determine effects of states do most of Rn D/R; fed. IAQ program; WY has most effort is hodge-podge: try dif. aggressive Rn D/R and best things results - even so, awareness level is 85% while testing level is 16% "',. 26 "
  27. 27. Appendix 3 WSDOH State Indoor Radon Grant Program Final Report: July 1, 1991 - June 30, 1992 • Report period represent Year 2 of EPA grant • Staffing during grant period was 7 FTE, plus 1 temporary • Participation on committees and symposiums • WSDOH convened meeting of state-wide Radon Task Force • Attended bimonthly meeting of SBCC tech. adv. gp. during 4-mo. period; attended Energy Code Comm. and 2 SBCC meetings • Advise key gps. re Rn • Clover Park S.D. Innovative Sch. Proj.: numerous meetings w/staff, 2 publ. mtgs.; presentation to ESD • Advised and made 1 presentation to CRPDC committees • 7 presentations to various conferences and committees • 3 presentations to state gov't bodies; aided in passage of legislation that requires local bldg. officials to supply testing device in new home construe. • Continued to augment and update database • Eliminated 1 fact sheet, updated 4 others • Updated telephone info service; 1,059 calls received during yr. - residential info was primary request, then basic Rn info and where to purchase test kits • Lending library has various materials, which were used by public and contractors • PSA's • Issued 3 press releases • Issued press releases and articles re Clover Park S.D. project • Published 4 articles in various newsletters • Rn promotion by Haggen Food Stores -- may have contributed to increase in calls (54 to 133) in Feb. • Various program staff attended at least 10 training sessions/conferences • Conducted testing survey and mail survey • Produced and distributed brochure for builders to distribute to new home occupants, and collaborated in brochure for local building officials to distribute to new home owners • Began, but never completed, project to send letters to new home owners re radar detector to be provided by builders (project superseded by new law requiring local officials to provide detector) • Gave contract money to NE WA Rural Resources Dev't Assoc. to conduct publ. info. program to test, diagnose, and mitigate in at least 100 econ. disadv'd homes in 3 counties (project never completed due to lack of matching funds from contractor); project converted to publ. info. effort through various media and presentations ' • CSRAT Project compiled materials, offered training re Rn in schools (113 total attendees plus 38 more in Clover Park S.D.) • 40 buildings te~ted in Clover Park 27 I

×