Using Rubrics to Enhance Extension Program Planning and Evaluation: A Health Insurance Literacy Example Bonnie Braun, PhD Professor, Department of Family ScienceSpecialist, University of Maryland Extension Faculty Scholar, Horowitz Center for Health Literacy University of Maryland firstname.lastname@example.org
Why Rubrics? Encourages consistent and thorough evaluation Assists in evaluation of existing and new programs, curricula and educational materials Permits multiple individuals to make judgments using common definitions
University of Maryland Health SmartHealth Insurance Literacy Initiative(HILI) Emergent area for education Strives to enhance the confidence and competence of consumers in making health insurance purchasing decisions Targets both those who already purchase insurance and those newly eligible for health insurance under the Affordable Care Act From Maryland-only to multi-state initiative Pilot testing launched April 1, 2013
Design Specifications &Products Comprehensive, systematic process* Health literacy principles Science Theory *Coleman, et al, 2011
Curriculum Assessment Tool Curriculum Assessment Tool -- provides a standardized set of criteria to evaluate existing educational curricula and to use in creating new educational curricula. Uses a four-point scale (Effective, Good, Fair, and Ineffective) Based on recommendations of Extension educators who provided a rationale for curricula review* *Coleman et al, (2011)
Educational Materials Tool Educational Materials Assessment Tool -- provides a standardized set of criteria to judge educational materials used in programs and in curricula. Can be used both for critiquing existing educational materials and creating new materials. Based on Doak et al (2007) which provides a standardized way to analyze materials for learners with varying levels of prose, document and quantitative literacy.
How did we create them? The role of theory… Both rubrics grounded in theoretical principles of adult learning, health literacy, health behavior WHY? Application of theory to educational interventions increases the likelihood that intended outcomes will be achieved; Theories provide the rationale for how the intervention is strategically structured and delivered; and Theories offer the basis for assessment of the program’s degree of success in achieving intended outcomes. Braun, et al (forthcoming)
How did we create them? The roleof theory… Depends on situation, targeted population, understanding of behavior change and determination of outcomes that are strategic, measurable, achievable, relevant and timely* Level of intervention—individual, family or other group, community or policy—will guide the appropriate programming theories. For the Health Insurance Literacy Initiative curriculum, we used individual-level theories: social cognitive theory, stages of change or readiness, theory of planned behavior, communications, adult or youth development, empowerment, and evaluation and action research *Doran, 1981
How did we create them? The role of theory… Key theoretical concepts can be combined into a model that frames the design and measurement of educational programming impact. The theories we selected directed us to: Involve the targeted population Identify the level of confidence, competence and sense of control before and after interventions. Create messages and deliver via appropriate channels Design learning experiences to increase critical thinking and reflection. Assure that evaluation of need, process and outcome is effectively conducted and reported.
How can you use these tools? Language of the rubrics could be substituted with appropriate vocabulary for the curricula content under review. May need revise based on the level of intervention (family, individual, community) and the intent or goals of the program. May need other theories to guide the development of programs and assessment tools if not focused on change-making.
Let’s practice…1. Find your copy of the Educational Materials Assessment Tool and the Scoring Sheet2. Review the provided document3. Complete the scoring sheet as it relates to the material
Questions for discussion: What are your thoughts about the material after using the rubric guidelines to assess it? In what ways did you find the rubric useful in guiding your evaluation of the material?
References Braun, B., McCoy, T., & Finkbeiner, N. (In press). Extension Education Theoretical Guide with Criterion- Referenced Assessment Tools. College Park: University of Maryland Extension Coleman, G., Byrd-Bredbenner, C., Baker, S., & Bowen, E. (2011). Best practices for extension curricula review. Journal of Extension, 49, Article 2T0T1. Available at: http://www.joe.org/joe/2011april/tt1.php Doak, C.C., Doak, L.G., & Root, J.H. (2007). Teaching patients with low literacy skills. (2nd ed.). Philadelphia: J.B. Lippincott Company. Doran, G. T. (1981). There’s a S.M.A.R.T. way to write management’s goals and objectives. Management Review, 70, 35-36.
AcknowledgementThis presentation and the rubrics were co-createdwith Nicole Finkbeiner, MS, GraduateResearch Assistant, University of MarylandDepartment of Family Sciences.