Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Gauging Effectiveness of Instructional Grants


Published on

Presentation at the POD Network Conference, Pittsburgh, PA, October 26, 2007

Published in: Economy & Finance, Education
  • If you download the slideshow, you'll find my notes pages included on each slide. As I said during the session, I welcome your comments and hope that those of us interested in better understanding the outcomes of grantmaking can engage in an ongoing sharing of ideas, evaluation methods, and tools.<br /><br/>
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Gauging Effectiveness of Instructional Grants

  1. 1. Gauging the Effectiveness of Instructional and Institutional Development Grants Lynda Milne Minnesota State Colleges & Universities Center for Teaching & Learning POD Network Conference, Pittsburgh October 26, 2007
  2. 2. Our center, like many, has a long history of awarding grants to faculty. In this session we will discuss two grant programs and the systems we developed to categorize and analyze data about the purposes, topic areas, activities, and outcomes of grant projects.
  3. 3. Acknowledgements David Laverny-Rafter CTL Faculty Project Specialist Professor, Political Science Minnesota State University, Mankato Thomas Wortman Assistant Director for Grant Programs Minnesota State Colleges & Universities Center for Teaching and Learning
  4. 4. Why Instructional Grants? <ul><li>Tradition (scholarship, research) </li></ul><ul><li>Foster innovation </li></ul><ul><li>Demonstration projects/ experiments/pilots </li></ul><ul><li>Allocate scarce resources </li></ul><ul><li>Competition fosters excellence </li></ul><ul><li>…(certainly many other) </li></ul>
  5. 5. How Little We Know <ul><li>Levinson-Rose & Menges, 1981 </li></ul><ul><ul><li>Frequency, popularity of programs </li></ul></ul><ul><li>Eble & McKeachie, 1985 </li></ul><ul><ul><li>Positive faculty perception </li></ul></ul><ul><li>Jacobsen, 1989 </li></ul><ul><ul><li>Little support from faculty </li></ul></ul><ul><li>Weimer & Lenze, 1991 </li></ul><ul><ul><li>Grants at 40% to 78% institutions </li></ul></ul><ul><ul><li>70% to 90% rated highly </li></ul></ul><ul><li>McAlpine & Gandell, 2003 </li></ul><ul><ul><li>Grants as SoTL; potential impact </li></ul></ul>
  6. 6. How Much We Do <ul><li>What teaching incentive grants do you manage? </li></ul><ul><li>What kinds of assessment requirements are included in your guidelines? </li></ul><ul><li>How do your grantees report to you? </li></ul><ul><li>How do you report to your campus/provost/others the benefits and outcomes of grants? </li></ul><ul><li>Are grants having a positive impact? </li></ul>
  7. 7. How Much We Do <ul><li>What teaching incentive grants do you manage? </li></ul><ul><li>What kinds of assessment requirements are included in your guidelines? </li></ul><ul><li>How do your grantees report to you? </li></ul><ul><li>How do you report to your campus/provost/others the benefits and outcomes of grants? </li></ul><ul><li>Are grants having a positive impact? </li></ul>
  8. 8. CTL Grantmaking History since 2000 <ul><li>Major Bush funds initiatives focused on active learning, 1999-2005 </li></ul><ul><ul><li>$2M for 200+ grants </li></ul></ul><ul><li>Systemwide teaching grants </li></ul><ul><ul><li>$350K for 56 projects </li></ul></ul><ul><li>Systemwide course redesign </li></ul><ul><ul><li>$250K for 11 grants </li></ul></ul><ul><li>Multiple outcomes </li></ul><ul><li>Multi-focused summative evaluations </li></ul><ul><li>http://www. ctl . mnscu .edu/programs/grants/ </li></ul>
  9. 9. Minnesota State Colleges & Universities System <ul><li>Merged public higher education system (all except University of Minnesota), 1995 </li></ul><ul><li>7 state universities </li></ul><ul><li>25 community/technical colleges </li></ul><ul><li>240,000 students </li></ul><ul><li>8,000+ faculty </li></ul><ul><li>Graduate majority of state’s teachers, nurses, “first responders” and law enforcement professionals </li></ul>
  10. 10. 2004 Legislature Targets Funds <ul><li>$6M per year for “competitive compensation for initiatives to promote excellence in student learning” </li></ul><ul><li>2005-2007 two new programs </li></ul><ul><ul><li>College Faculty Awards for Excellence (individual) </li></ul></ul><ul><ul><li>Initiatives to Promote Excellence in Student Learning (institutional) </li></ul></ul>
  11. 11. Guidelines <ul><li>Efforts above and beyond regular work responsibilities </li></ul><ul><li>Focus on student learning </li></ul><ul><li>Alignment with strategic goals </li></ul><ul><li>Work plan themes/critical targets </li></ul><ul><ul><li>Reading and writing (transitions—first year and entry into major) </li></ul></ul><ul><ul><li>STEM outcomes </li></ul></ul><ul><ul><li>Critical thinking </li></ul></ul>
  12. 12. Strategic Directions <ul><li>Increase access and opportunity </li></ul><ul><li>Promote and measure high-quality learning programs and services </li></ul><ul><li>Serve state and regional economic needs </li></ul><ul><li>Innovate for current and future educational needs and efficiency </li></ul>
  13. 13. Work Plan <ul><li>Eliminate the achievement gap </li></ul><ul><li>Provide universal access from pre-kindergarten through the first two years of college </li></ul><ul><li>Promote increased participation in science, technology, engineering and math, known as STEM, fields </li></ul><ul><li>Expand corporate learning </li></ul><ul><li>Develop colleges and universities for Minnesota's future </li></ul><ul><li>Build organizational capacity for change and innovation </li></ul>
  14. 14. 2005 Faculty Development Survey <ul><li>Critical thinking topic of highest interest </li></ul>
  15. 15. Survey says…
  16. 16. Course Outcomes: 2004-05 DFW Rates <ul><li>First-semester college writing courses: 10% – 30% </li></ul><ul><li>Developmental reading and writing courses: 25%– 54% </li></ul><ul><li>College algebra: 15% - 58% </li></ul><ul><li>Developmental math: 15% - 64% </li></ul><ul><li>Physical sciences: 12% – 48% </li></ul><ul><li>Source: Analysis of large-course (5+ sections) grade distributions, Fall 2004 – Spring 2005. Ranges across 32 institutions. </li></ul>
  17. 17. Course Outcomes: 2004-05 DFW Rates Courses with 5 or More Sections 20.9 27.3 40.1 25.6% 2-Year Colleges 11,935 18,244 46,346 91,162 # Students Enrolled All Sections 20.2 Physical Sciences 19.4 Biological/ Life Sciences 32.9 Mathematics 12.1% English and Speech 4-Year Univs
  18. 18. Structuring Guidelines <ul><li>Review the excerpts of the guidelines for each of the two programs. </li></ul><ul><li>Then on the “How Would You Assess These Grants?” handout, answer the questions that we faced about evaluating and reporting on these grants. </li></ul>
  19. 19. Reporting Aligned with Purposes <ul><li>Outcomes </li></ul><ul><li>Principles </li></ul><ul><li>Strategies </li></ul><ul><li>Dissemination </li></ul><ul><li>Sustainability </li></ul>
  20. 20. Analyzing the Reports <ul><li>Description </li></ul><ul><ul><li>How many, what proportion of grant managers </li></ul></ul><ul><ul><ul><li>Improved learning </li></ul></ul></ul><ul><ul><ul><li>Created new curriculum </li></ul></ul></ul><ul><ul><ul><li>Innovated in strategic areas </li></ul></ul></ul><ul><li>Analysis </li></ul><ul><ul><li>Trends </li></ul></ul><ul><ul><li>Relationships </li></ul></ul><ul><li>Reporting </li></ul><ul><ul><li>Impact </li></ul></ul>
  21. 21. Outcome-Based Evaluation <ul><li>Description </li></ul><ul><ul><li>Inputs </li></ul></ul><ul><ul><li>Outputs </li></ul></ul><ul><ul><li>Outcomes </li></ul></ul><ul><li>Analysis </li></ul><ul><ul><li>Trends </li></ul></ul><ul><ul><li>Relationships </li></ul></ul><ul><li>Reporting </li></ul><ul><ul><li>Impact </li></ul></ul>
  22. 22. Influences shaping our evaluation work <ul><li>Scholarship Assessed </li></ul><ul><li>NCAT </li></ul><ul><li>Logic Model/Outcome- Based Evaluation methods </li></ul>
  23. 23. Scholarship Assessed <ul><li>Scholarship should be the outcome we measure </li></ul><ul><li>Pre-grant (formative assessment) </li></ul><ul><ul><li>Clear goals </li></ul></ul><ul><ul><li>Adequate preparation </li></ul></ul><ul><ul><li>Appropriate methods </li></ul></ul><ul><li>Post-grant (summative </li></ul><ul><ul><li>Significant results </li></ul></ul><ul><ul><li>Effective presentation </li></ul></ul><ul><ul><li>Reflective critique </li></ul></ul><ul><ul><ul><li>- Glassick, Huber, Maeroff (1997). Scholarship Assessed </li></ul></ul></ul>
  24. 24. NCAT Course Redesign <ul><li>NCAT ‘s Program in Course Redesign showed clearly 3 big outcomes from a major grant project: </li></ul><ul><li>Improvements in learning </li></ul><ul><ul><li>Higher grades </li></ul></ul><ul><ul><li>Better test scores </li></ul></ul><ul><ul><li>Assessments of learning quality/ depth </li></ul></ul><ul><li>Lower DIW rates </li></ul><ul><li>Lower instructional costs </li></ul>
  25. 25. Outcomes-Based Evaluation Methods <ul><li>Inputs (resources) </li></ul><ul><li>Outputs (activities, products) </li></ul><ul><li>Outcomes (results of the outputs, changes that occur--in learning, attitudes, behavior, etc.) </li></ul><ul><li>Impact (long-term benefits of the program) </li></ul>
  26. 26. Applicability <ul><li>Online database-driven proposal and reporting (at least reporting) software </li></ul><ul><ul><li>Easier for grantees </li></ul></ul><ul><ul><li>Standardizes data collection </li></ul></ul><ul><ul><li>Enables data categorization and capture </li></ul></ul><ul><li>Grants software packages available </li></ul><ul><ul><li>– Intelligrants and other “big program” grants management applications </li></ul></ul><ul><li>At a minimum: a system for analyzing, categorizing according to: </li></ul><ul><ul><li>Purpose </li></ul></ul><ul><ul><li>Principles, themes, emphases </li></ul></ul><ul><ul><li>Strategies, activities </li></ul></ul><ul><ul><li>Outcomes </li></ul></ul>
  27. 27. Grants and SoTL <ul><li>Faculty developers can: </li></ul><ul><li>Encourage scholarly approach es to conducting teaching grant projects </li></ul><ul><li>Develop a scholarship of grants evaluation </li></ul><ul><ul><li>Formative and summative assessment methods </li></ul></ul><ul><ul><li>Establishment of some common measures </li></ul></ul><ul><ul><li>Reporting of trends and conclusions </li></ul></ul>
  28. 28. <ul><li>Thanks. I look forward to continuing the conversation! </li></ul><ul><li>Please contact me or Thomas Wortman if you need any information or examples, and please share yours with us. </li></ul>