How to Improve Extension
Program Outcomes using a
Monitoring and Evaluation
Strategy
Dr. Ayanava Majumdar
Extension Entomo...
Focus of this presentation
• Structure & implementation of Alabama IPM
Program
• Provide some basics of evaluation – Exten...
About the Author (Ayanava)
Ext. evaluations:
• Memberships: American Eval. Assoc., Southeast Eval. Assoc.
• AEA365 Blog Cu...
Alabama Small Farms
• Produce fruits & vegetables that
are consumed locally
• Area = 6,000+ acres
• Crop value = $48 milli...
Alabama Cooperative Extension System
Ref.: Robinson, Dubois, Bailey (2005). Journal of Extension.
Regional Extension Agent...
ACES Commercial Horticulture Team –
Vegetable IPM Team Members
ACES Home Grounds Team – Vegetable IPM
Team Members
Chris Becker, REA Willie Datcher, REA Mike McQueen, REA
Alfred Jackson...
Awards & Recognition
• ‘Friends of IPM - Pulling Together Award’ from the Southern
Region IPM Center, Raleigh, NC (2014) f...
Structure of Alabama IPM Program:
A Model Program for Study
Fund acquisition (external)
• Funds needed to support REA travel & IPM
demonstration plots
• USDA/NIFA Collaborative Grant...
Theory of Planned Behavior (TpB)
Ajzen, I. 1991. The Theory of Planned Behavior. Organizational Behavior and Human
Decisio...
Theory of Change: Ext. Logic Model
IPM Theory of
Change
Communities
ImpactsModified from Kokate et al. 2009
Farmer-to-farmer
Train-the-trainer
Sustain change...
Ext. IPM Project Implementation
Continuous needs
& outcomes
assessments
1
2
3
4
1
2
3
4
Model by Dr. A. Majumdar 2011
Base...
CommunicationStrategy(IPM-CORE)
No cost tools
Technology identified & assessed
Story developed
Press release
Extension Web...
Level 1: Regional IPM Meetings
• Participants: Vegetable producers, certified
organic farms, gardeners & homeowners, crop
...
Regional IPM Training Meetings (Level 1 training)
Level 2: IPM Workshops
• Participants: Small vegetable producers, low
resource farmers, Master Gardeners, nonprofit
organi...
Hands-on IPM Workshops (Level 2 training)
Level 3: IPM Field Days
(Demonstration Sites)
• Participants: Producers, low resource farmers,
gardeners
• Teaching techni...
IPM Field Days (Level 3 training)
Farmer-to-farmer training model is
very influential for IPM project
Farmer-to-Farmer training
during IPM Field Days
Causes high information retention &
impact on communities
Basics of Program Monitoring &
Evaluation (PME)
Evaluation
• Evaluation is the systematic collection of information
about the merit, worth or significance of a program
(S...
Rate your IPM program evaluation experience
(useful exercise for audience)
Common myths!
• My clients like me, so I do not
need to evaluate.
• Evaluation is hard & outside my
job description.
• Pub...
Who can do the evaluation?
Internal evaluators
• Project leadership team
directly guides evaluation
(cost-effective)
• Ind...
CDC Framework for Program Evaluation Standards
Thomas Chapel, Chief Eval. Officer, CDC
Purpose of Evaluation
Formative evaluation
• Done when the program
is active
• Project monitoring
• Continuous use
• Flexi...
Utilization-focused Evaluations
• Evaluation technique emphasizing USE (Patton 1993)
• Improvement-oriented evaluation: in...
Evaluation techniques
Reactive assessment:
• Participants are aware of the
assessment
• Examples: surveys, interviews,
tes...
Choose outcome ‘indicators’ carefully
• Indicators are critical questions
• Based on program objectives
• Used for monitor...
Taylor-Powell & Henert 2008
Needs
assessments
Process
evaluations
(formative)
Outcome
Evaluations
(formative)
Impact
evaluations
(summative)
Project
m...
What do we measure?
Hierarchy of Outcome Effects
• Level 1: REACTION
• Level 2: LEARNING
• Level 3: BEHAVIOR
• Level 4: RE...
Hierarchy of Effects in IPM Program
(In Pictures)
• Level 1: REACTION
• Level 2: LEARNING
Hierarchy of Effects in IPM Program
(In Pictures)
• Level 3: BEHAVIOR
• Level 4: RESULTS
AL IPM Project
OUTCOMES
(Usefulness of Project
Monitoring)
Project outcomes (2008-2012)
• Developed a unique Theory of Change and
Transformational Extension Education Model based
on...
• Rising interest of producers/Pull on project:
– Over 10 times rise in participation in IPM events
– 10 times rise in num...
IPM Theory of
Change
Communities
ImpactsModified from Kokate et al. 2009
Farmer-to-farmer
Train-the-trainer
Sustain change...
67
259
505
945
39
54
57 55
3 4 28 270
100
200
300
400
500
600
700
800
900
1000
2009 2010 2011 2012
Total participants
Surv...
1.5
6 6.9
73
36
26
5
8 10.18
34
29.1
11
7
20.9
0
10
20
30
40
50
60
70
80
2009 2010 2011
Beginning farmers (%)
Conventional...
28
13.6
9.3
0
5
10
15
20
25
30
2009 2010 2011
Average land holding (acres)
CHANGING NATURE OF EXTENSION CLIENTELE
(reason ...
Conventional vegetable farmers = 15-50 acres
Transitioning farmers = 10-14 acres
Beginning farmers = <10 acres
Low resourc...
45
29
10
23.2
30
2.2
20
11.4
2.5
5
0
5
10
15
20
25
30
35
40
45
50
2008 2011
Lack of awareness (%)
High cost (%)
Difficult ...
PROJECT IMPACTS
(Watch 2013 Impact Video at
http://youtu.be/aqrjQINLUdw)
Impact Evaluation, 2010-2011 (n=58):
• E-subscriptions growth rate = 12%
• Current subscription base ~1350 with much highe...
Program Impacts: IPM Adoption
Indicator 2011 2010 2009 Increase
Adoption of insect
monitoring/scouting
practices
73.5% 41....
Economic impacts
• Yield loss in the absence of IPM: 44% or
more
• Vegetable producers gained $246 per acre by
using IPM r...
• Increased use of biological insecticides
(42%)
• Conservation of natural enemies
(unknown)
• Profits occur by judicious ...
• Three new partnerships with influential
commodity & consumer organizations
• New partnerships with nonprofit
agencies se...
Charles Brannon, vegetable producer (Addison, AL): “This is the first year we used
the IPM recommendation from the handboo...
Danny Dickie, vegetable producer (Oneonta, AL): “We use the
insecticide recommendations from the SE Vegetable Handbook
and...
Albert Riddle, vegetable producer (Titus, AL): “We use the integrated
pest management recommendations provided by Chip Eas...
AND it all comes down to writing good
reports…
Questions for Dr. A?
Thank you for your patience!
Upcoming SlideShare
Loading in …5
×

How to Increase Extension Program Outcomes and Impacts (Slides for Educators)

727 views

Published on

This video is meant for Extension educators to demonstrate the feasibility and usefulness of an Extension program and Evaluation Strategy that is based on specific goals. This presentation is a basic version and we have much more information that is part of continuous improvement in the slideshow. We will share other presentations with more information - so this is just the beginning! For program evaluation/monitoring questions, call 251-331-8416 or email bugdoctor@auburn.edu. For looking at some of my IPM Program evaluation publications, visit www.aces.edu/go/87 and click on 'IPM Evaluation Toolkit' in the menu. Thank you.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
727
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
29
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

How to Increase Extension Program Outcomes and Impacts (Slides for Educators)

  1. 1. How to Improve Extension Program Outcomes using a Monitoring and Evaluation Strategy Dr. Ayanava Majumdar Extension Entomologist, SARE Coord. (AU) 111-A Duncan Hall, Auburn University bugdoctor@auburn.edu 251-331-8416
  2. 2. Focus of this presentation • Structure & implementation of Alabama IPM Program • Provide some basics of evaluation – Extension perspective! • Encourage you to be consistent in your evaluation efforts • Develop capacity building within organization
  3. 3. About the Author (Ayanava) Ext. evaluations: • Memberships: American Eval. Assoc., Southeast Eval. Assoc. • AEA365 Blog Curator (2010-2011), Organization Capacity Building TIG Grant Panel, Ext. Eval. TIG member • Lead evaluator: USDA-NIFA, AL DoA, commodity grants, School IPM, AACAAS, NACAA Organizational capacity building: • Initiated ACES Program Evaluation Resource Committee, 2010… created the Evaluation Toolkit [online] • Workshops, webinars, publications: total ~7 hr, trained 161 Agents, County Coordinators, Specialists, Ext. Administrators
  4. 4. Alabama Small Farms • Produce fruits & vegetables that are consumed locally • Area = 6,000+ acres • Crop value = $48 million • Growth rate for industry = 14% • Vegetables: Tomatoes, sweet corn, watermelon, crucifers • Farmer markets: 135 (1100 farms) • Direct sales from AL farmers to consumers account for 0.2% of farm sales (Meter 2012)
  5. 5. Alabama Cooperative Extension System Ref.: Robinson, Dubois, Bailey (2005). Journal of Extension. Regional Extension Agents (REAs) Regional or statewide IPM events Outputs change annually Outcomes do not change! Impacts now underway. + CECs
  6. 6. ACES Commercial Horticulture Team – Vegetable IPM Team Members
  7. 7. ACES Home Grounds Team – Vegetable IPM Team Members Chris Becker, REA Willie Datcher, REA Mike McQueen, REA Alfred Jackson, Tuskegee Extension
  8. 8. Awards & Recognition • ‘Friends of IPM - Pulling Together Award’ from the Southern Region IPM Center, Raleigh, NC (2014) for IPM newsletter. • Friends of IPM: Future Leader Award (2012) By Southern Region IPM Center, Raleigh, NC Recognizes excellence in IPM program directly • Search for Excellence – Crops (2012) By National Association of County Agricultural Agents Recognizes a high impact IPM program • Achievement Award (2012) By the National Association of County Agricultural Agents • Communication Awards (6) By National Association of County Agricultural Agents
  9. 9. Structure of Alabama IPM Program: A Model Program for Study
  10. 10. Fund acquisition (external) • Funds needed to support REA travel & IPM demonstration plots • USDA/NIFA Collaborative Grants: $124,500 (part of two large grants worth $2.1 million) = 6% • SARE support: $10,000 + personnel for website • Alabama Department of Agriculture: $25,000 • Industry support: $45,000
  11. 11. Theory of Planned Behavior (TpB) Ajzen, I. 1991. The Theory of Planned Behavior. Organizational Behavior and Human Decision Processes. 50: 179-211
  12. 12. Theory of Change: Ext. Logic Model
  13. 13. IPM Theory of Change Communities ImpactsModified from Kokate et al. 2009 Farmer-to-farmer Train-the-trainer Sustain change Key farmers Technology acceptance ExtensionResearch Project Management Capacity Building Need identified (context)
  14. 14. Ext. IPM Project Implementation Continuous needs & outcomes assessments 1 2 3 4 1 2 3 4 Model by Dr. A. Majumdar 2011 Based on Rockwell et al. 2003
  15. 15. CommunicationStrategy(IPM-CORE) No cost tools Technology identified & assessed Story developed Press release Extension Websites Social sharing (Slideshare, YouTube) Social networking (Facebook) Medium cost tools IPM Communicator Newsletter Extension regional meetings, IPM Web Conferences IPM Project flyers, bookmarks High cost tools Multiple partnerships (~20) IPM Exhibitions, Newspaper advertisements IPM on-farm demonstrations/ communications Small Farm IPM Field Guide IPM App?
  16. 16. Level 1: Regional IPM Meetings • Participants: Vegetable producers, certified organic farms, gardeners & homeowners, crop advisors, nonprofit organizations (partners) • Teaching techniques: Presentation, pest specimens • Typical length of talks: 46 min • Participants total: 1,535 • Number of events completed: 58 • Evaluation: Events are monitored for quality and impacts via multimodal surveys
  17. 17. Regional IPM Training Meetings (Level 1 training)
  18. 18. Level 2: IPM Workshops • Participants: Small vegetable producers, low resource farmers, Master Gardeners, nonprofit organizations, Extension Agents (in-service) • Teaching techniques: Case studies, simulations, specimens • Typical length of talks: 140 min • Participants total: 570 • Number of events completed: 14 • Evaluation: Qualitative & quantitative feedback, impact evaluations
  19. 19. Hands-on IPM Workshops (Level 2 training)
  20. 20. Level 3: IPM Field Days (Demonstration Sites) • Participants: Producers, low resource farmers, gardeners • Teaching techniques: Scouting, pest identification, decision-making system • Typical training time: 3-4 hours • Participants total: 308 • Number of events completed: 12 • Evaluation: Qualitative & quantitative feedback, impact evaluations, farm visits
  21. 21. IPM Field Days (Level 3 training)
  22. 22. Farmer-to-farmer training model is very influential for IPM project
  23. 23. Farmer-to-Farmer training during IPM Field Days Causes high information retention & impact on communities
  24. 24. Basics of Program Monitoring & Evaluation (PME)
  25. 25. Evaluation • Evaluation is the systematic collection of information about the merit, worth or significance of a program (Scriven 1999). • Evaluation is broadly defines as the systematic collection of data – both quantitative and qualitative – to aid users in developing knowledge about & managing a targeted set of activities (Scheirer 2012). • Ext. evaluation practice aims to find plausible conclusions, not cause & effect!
  26. 26. Rate your IPM program evaluation experience (useful exercise for audience)
  27. 27. Common myths! • My clients like me, so I do not need to evaluate. • Evaluation is hard & outside my job description. • Public does not respond to my surveys. • Nobody is going to look at the evaluations once they are done. • Only ‘experts’ can do evaluation. Why bother! PERCEPTION REALITY • Evaluation helps document your success. • 20% effort generates 80% information. • Provide time to respond, part of agenda, show improvements! • Share the results with all stakeholders. • Evaluation can be done by any trained individual committed to the standards.
  28. 28. Who can do the evaluation? Internal evaluators • Project leadership team directly guides evaluation (cost-effective) • Indicates organizational commitment /capacity building • Project monitoring possible (utilization-focus) • Fast assessment of reactive programs • Good participation External evaluators • Neutral third-party observers (expensive) • No internal capacity building • Good for impacts • Slow assessments & less useful • Poor participation (trust) Based on Patton (1997)
  29. 29. CDC Framework for Program Evaluation Standards Thomas Chapel, Chief Eval. Officer, CDC
  30. 30. Purpose of Evaluation Formative evaluation • Done when the program is active • Project monitoring • Continuous use • Flexible (mixed) designs Summative evaluation • Done at the end of a program • Measure impacts • One time use • Rigid design, expensive Patton (1997), Scriven (1972)
  31. 31. Utilization-focused Evaluations • Evaluation technique emphasizing USE (Patton 1993) • Improvement-oriented evaluation: includes formative evaluations, quality enhancement, Total Quality Management (TQM) • TQM includes using information systems to monitor program efforts & outcomes continuously • Use feedback as a monitoring system (like insect traps!). + Trouble= = Do something about it!
  32. 32. Evaluation techniques Reactive assessment: • Participants are aware of the assessment • Examples: surveys, interviews, tests Non-reactive assessment: • Participants are unaware of the assessment • Examples: observation Haas (2005)
  33. 33. Choose outcome ‘indicators’ carefully • Indicators are critical questions • Based on program objectives • Used for monitoring progress • Usually expressed as rates, percentages, efficiencies, etc. • There should be few trackable indicators (QUAL and QUAN)
  34. 34. Taylor-Powell & Henert 2008
  35. 35. Needs assessments Process evaluations (formative) Outcome Evaluations (formative) Impact evaluations (summative) Project monitoring Project improvement Modified strategy Modified outputs Evaluation Strategy in Extension IPM Program Graphic by Dr. Ayanava Majumdar, ACES
  36. 36. What do we measure? Hierarchy of Outcome Effects • Level 1: REACTION • Level 2: LEARNING • Level 3: BEHAVIOR • Level 4: RESULT Increasing complexity of evaluation Evaluate during program Kirkpatrick Four Levels, 1959, 1994 Post program evaluation QUAN QUAL
  37. 37. Hierarchy of Effects in IPM Program (In Pictures) • Level 1: REACTION • Level 2: LEARNING
  38. 38. Hierarchy of Effects in IPM Program (In Pictures) • Level 3: BEHAVIOR • Level 4: RESULTS
  39. 39. AL IPM Project OUTCOMES (Usefulness of Project Monitoring)
  40. 40. Project outcomes (2008-2012) • Developed a unique Theory of Change and Transformational Extension Education Model based on long-term outcomes • New producers reached 14% and rising (transitioning farms 7-10%). • General IPM adoption rate: over 90% • Adoption rate for unique IPM tactics: 63% (up from 38%) • Publication in print & electronic have significantly reduced barrier to IPM adoption • Funding for Small Farm IPM program has doubled since 2012. Much focus on Ext. Agent training.
  41. 41. • Rising interest of producers/Pull on project: – Over 10 times rise in participation in IPM events – 10 times rise in number of IPM events • Barriers to IPM adoption (crop advisors): – Lack of awareness reduced by 33% in 3 years – Difficulty in accessing information reduced by 33% • Barriers to IPM adoption (producers): – Lack of awareness reduced 16% in 3 years – Difficulty in accessing IPM information reduced 13% Project outcomes (2008-2012)
  42. 42. IPM Theory of Change Communities ImpactsModified from Kokate et al. 2009 Farmer-to-farmer Train-the-trainer Sustain change Key farmers Technology acceptance ExtensionResearch Project Management Capacity Building Need identified (context)
  43. 43. 67 259 505 945 39 54 57 55 3 4 28 270 100 200 300 400 500 600 700 800 900 1000 2009 2010 2011 2012 Total participants Survey return rate (%) Major IPM Events GROWING INTEREST OF FARMERS IN IPM
  44. 44. 1.5 6 6.9 73 36 26 5 8 10.18 34 29.1 11 7 20.9 0 10 20 30 40 50 60 70 80 2009 2010 2011 Beginning farmers (%) Conventional producers (%) LRFs (%) Transitioning farms (%) Gardeners (%) NATURE OF AUDIENCE AT IPM EVENTS
  45. 45. 28 13.6 9.3 0 5 10 15 20 25 30 2009 2010 2011 Average land holding (acres) CHANGING NATURE OF EXTENSION CLIENTELE (reason for splitting IPM program into two campaigns to increase impacts) Organic/ Small Farms IPM Campaign Conventional farmers (with 15 to 50 acres) 2012 Farmers (<10 acres) Gardeners (<2 acres) Conventional/ Large Farms IPM Campaign
  46. 46. Conventional vegetable farmers = 15-50 acres Transitioning farmers = 10-14 acres Beginning farmers = <10 acres Low resource farmers = 2 to 4 acres CHANGING NATURE OF EXTENSION CLIENTELE (reason for splitting IPM program into two campaigns to increase impacts) Two IPM Campaigns launched in 2012 with separate strategies and outcomes. Rapid increase in small farm IPM funding and number of participants due to rising interest in local food systems.
  47. 47. 45 29 10 23.2 30 2.2 20 11.4 2.5 5 0 5 10 15 20 25 30 35 40 45 50 2008 2011 Lack of awareness (%) High cost (%) Difficult to access information (%) Low availability of insecticides (%) Lack of time (%) BARRIERS TO IPM ADOPTION & CORRECTIONS Major concerns at the time
  48. 48. PROJECT IMPACTS (Watch 2013 Impact Video at http://youtu.be/aqrjQINLUdw)
  49. 49. Impact Evaluation, 2010-2011 (n=58): • E-subscriptions growth rate = 12% • Current subscription base ~1350 with much higher readership. • 34% farmers, 28% company representatives, 38% others • 53% read for 15 minutes and 22% for 30+ minutes • 94% support continuation of the newsletter (2011 survey) • 70% use the information (2013 survey) • Six financial gains reports (2013): $3,550 in pesticide saving • Short-term impact of publication: $591 per case of adoption IPM Newsletter Impacts
  50. 50. Program Impacts: IPM Adoption Indicator 2011 2010 2009 Increase Adoption of insect monitoring/scouting practices 73.5% 41.0% +32% Adoption of insecticide recommendations 79.6% 46.5% 33% Use of biological insecticides 42.3% 26.5% 16% Overall IPM adoption 62.7% 38.5% 24% Economic & social impact assessments ongoing in 2012 and beyond. Received IRB Approval for Surveys.
  51. 51. Economic impacts • Yield loss in the absence of IPM: 44% or more • Vegetable producers gained $246 per acre by using IPM recommendations • Profits occur by judicious use of pesticides, adoption of appropriate control tactics after pest identification, and timely action. • Weekly IPM newsletter resulted in saving of over $500 per adoptive farmer
  52. 52. • Increased use of biological insecticides (42%) • Conservation of natural enemies (unknown) • Profits occur by judicious use of pesticides, adoption of appropriate control tactics after pest identification, and timely action. Environmental impacts
  53. 53. • Three new partnerships with influential commodity & consumer organizations • New partnerships with nonprofit agencies serving LRFs in Black Belt of Alabama • Impact assessments continue in 2012 & beyond. Social impacts
  54. 54. Charles Brannon, vegetable producer (Addison, AL): “This is the first year we used the IPM recommendation from the handbook and received training from Dr. A. In the hoop house alone, we sold about $5,500 worth of tomatoes which is double the output from last year. Fruitworms and stinkbugs used to do about 50% yield loss but not this year. With abundant production, we are taking our produce to large farmer markets in Birmingham and Decatur, not limiting ourselves just to the farm stand.” (Surveyed on July 5, 2012)
  55. 55. Danny Dickie, vegetable producer (Oneonta, AL): “We use the insecticide recommendations from the SE Vegetable Handbook and consult Mel Wade before we make a treatment decision. Without insecticides, we can potentially lose over 80% of our crops to worms and stink bugs that may result in over $20,000 loss per acre.” (Surveyed on May 30, 2012)
  56. 56. Albert Riddle, vegetable producer (Titus, AL): “We use the integrated pest management recommendations provided by Chip East – our Regional Extension Agent. I can lose over 50% of my tomato crop if I did not follow IPM for insect and disease management. That is about $10,000 loss per acre.” (Surveyed on May 23, 2012)
  57. 57. AND it all comes down to writing good reports… Questions for Dr. A? Thank you for your patience!

×