Building an Evaluation Model for Online HBSE Social Work Courses


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Building an Evaluation Model for Online HBSE Social Work Courses

  1. 1. James Young, MSW, MPA Mary Secret, Ph.D. Angie Mann-Williams, LCSW Virginia Commonwealth University School of Social Work
  2. 2. <ul><li>Brief review of the evaluation literature </li></ul><ul><li>Methodology/ Model Plan </li></ul><ul><ul><li>Evaluation Design </li></ul></ul><ul><ul><li>Student Learning Outcomes Questionnaires (SLOQ) </li></ul></ul><ul><ul><li>Resources </li></ul></ul><ul><li>Results </li></ul><ul><li>Implications </li></ul>
  3. 3. <ul><li>Over 4.6 million students took at least 1 online course in 2008. </li></ul><ul><ul><li>More than 1 in 4 higher ed. students now take at least one course online. (Allen & Seaman, 2010) </li></ul></ul><ul><li>From Meta-Analysis of Online Learning: </li></ul><ul><ul><li>On average, students in online learning conditions performed better than those receiving face-to-face instruction. (Bernard et al., 2004; U. S. Department of Education, Office of Planning, Evaluation, and Policy Development, 2009). </li></ul></ul>
  4. 4. <ul><li>“ Course and program outcomes achieved in social work distance education courses and programs are comparable to those achieved in traditional face-to-face programs”. (Macy, Rooney, Hollister, Freddolino, 2001). </li></ul><ul><li>No differences on educational outcomes or satisfaction. (Wilke & Vinton, 2006). </li></ul>
  5. 5. <ul><li>No significant differences between Online or Face-to-Face groups in Practice, Policy, and HBSE courses. (Seabury, 2005; Siebert, Siebert, & Spaulding-Givens, 2006; Woehle, Quinn, 2009; York, 2008). </li></ul>
  6. 6. <ul><li>Student socialization to the profession? </li></ul><ul><li>Online course design and evaluation? </li></ul><ul><ul><li>Content </li></ul></ul><ul><ul><li>Pedagogy </li></ul></ul><ul><li>Follow up with Alumni? </li></ul><ul><li>Evaluating student’s technical expertise? </li></ul><ul><ul><li>Readiness for online learning and Success </li></ul></ul><ul><li>Resources </li></ul><ul><ul><li>Technology </li></ul></ul><ul><ul><li>Time </li></ul></ul><ul><ul><li>Challenge with the rapidly changing nature of technology & trying to evaluate online social work courses. </li></ul></ul>
  7. 7. <ul><li>Explore differences and similarities between online/hybrid courses and face-to-face courses in the following areas: </li></ul><ul><li>Achievement of student learning outcomes as informed by selected EPAS Competencies </li></ul><ul><li>Inputs / Resources </li></ul><ul><ul><ul><ul><ul><li>Instructor time </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Type of Technology resources </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Type of Class room activities and assignments </li></ul></ul></ul></ul></ul><ul><li>Student Perspectives </li></ul>
  8. 8. <ul><li>IRB approved </li></ul><ul><li>Post-test only, quasi-experimental research design </li></ul><ul><li>Instructors in three sections of HBSE course agreed to participate in study </li></ul><ul><ul><li>One online course taught by experienced faculty (19 students) served as experimental group </li></ul></ul><ul><ul><li>One face-to-face course taught by doctoral student who served as teaching assistant for the online course (20 students) served as comparison group </li></ul></ul><ul><ul><li>Another face-to-face course taught by an experienced adjunct (19 students) served as second comparison group </li></ul></ul>
  9. 9. <ul><li>All three courses </li></ul><ul><li>followed the same basic syllabus, </li></ul><ul><li>used the same textbook (Hutchison), </li></ul><ul><li>used the same standardized power point slides that accompanied the textbook </li></ul><ul><li>had the same assignments </li></ul>
  10. 10. <ul><li>Online , 3 in-class meetings (begin, middle, and end of semester) </li></ul><ul><li>primarily weekly graded collaborative learning groups on Blackboard </li></ul><ul><li>Face to Face , weekly in-class meetings </li></ul><ul><li>primarily lecture, class discussion, and small group work in classroom </li></ul>
  11. 11. <ul><li>The online course and one of the face-to-face courses used the same collaborative learning group model, including the use of the same weekly group discussion questions </li></ul><ul><li>The online course instructor served as mentor for the face-to-face courses taught by an adjunct faculty and a doctoral student </li></ul><ul><li>The doctoral student served as teaching assistant for the online course </li></ul>
  12. 13. * from data collected on the Learning Outcomes Instrument
  13. 14. <ul><li>More part-time students self-selected into online class </li></ul>
  14. 15. <ul><li>Four vignettes (case studies), each focusing on a different life stage </li></ul><ul><ul><li>Case Study 1: Pregnancy, Early Childhood, Family Life Stage </li></ul></ul><ul><ul><li>Case Study 2: Late Adulthood, Infancy and Toddlerhood </li></ul></ul><ul><ul><li>Case Study 3: Adolescence </li></ul></ul><ul><ul><li>Case Study 4: Middle Childhood AND Young Adulthood </li></ul></ul>
  15. 16. <ul><li>10 – 12 questions per case study, developed to reflect the following EPAS teaching competency and practice behaviors: </li></ul><ul><li>Educational Policy 2.1.7 — Apply knowledge of human behavior and the social environment. </li></ul><ul><ul><li>P.B 2.1.7 a Utilize Conceptual Frameworks To Guide The Processes Of Assessment, Intervention, And Evaluation. </li></ul></ul><ul><ul><li>P.B 2.1.7b Critique And Apply Knowledge To Understand Person And Environment </li></ul></ul><ul><li>Educational Policy 2.1.4 – Engage Diversity And Difference In Practice </li></ul>
  16. 17. <ul><li>Each question identified for focus on different area relative to EPAS: </li></ul><ul><li>C -- utilization of conceptual framework </li></ul><ul><li>E -- critique and application of evidence-based knowledge </li></ul><ul><li>D -- engagement to diversity and difference </li></ul>
  17. 18. <ul><li>(c ) Rosa, Emma and Marie’s stories unfold as they progress through life. The term used to understand this relationship between time and human behavior and examines how chronological age, relationships, common life transitions, and social change shape people’s lives from birth to death. </li></ul><ul><li>Systems theory </li></ul><ul><li>Life course perspective </li></ul><ul><li>Cohort Study </li></ul><ul><li>Developmental psychology </li></ul><ul><li>none of the above is an appropriate term </li></ul>
  18. 19. <ul><li>(D) A social worker should be careful about applying Piaget’s theory of development when working with Maria because: </li></ul><ul><li>Maria is at the autonomy vs shame level of psychosocial development </li></ul><ul><li>Maria’s mother, Aida, was a teen parent who smoked </li></ul><ul><li>The theory does not account for variation in culture, family, and significant relationships on cognitive development </li></ul><ul><li>Maria has no contact with her father and he might want to remove her from the Surez family after her mother Aida was lost in the 9/11 tragedy </li></ul><ul><li>All of the above are equally important factors to consider when applying Piaget’s theory </li></ul>
  19. 20. <ul><li>(e) One of the best evidence-based strategies to reduce the risk factors and increase the protective factors for Tiffany’s 3 and 5 year old sons at this point in their lives is: </li></ul><ul><ul><li>Attendance at bi-weekly church services </li></ul></ul><ul><ul><li>Enrollment in a high quality early childhood education program </li></ul></ul><ul><ul><li>Parenting classes for Tiffany and Ruby </li></ul></ul><ul><ul><li>Have Tiffany marry the father of her children </li></ul></ul><ul><ul><li>All of the above would be equally effective strategies for the two boys </li></ul></ul>
  20. 21. <ul><li>The instrument was administered during the last class session of the semester </li></ul><ul><li>The instructors participating were provided a script to read to students participating in the evaluation </li></ul><ul><li>The instructors collected the instruments and placed them in sealed envelopes. </li></ul><ul><li>The envelopes were given to a graduate research assistant ensuring that the students information remained confidential </li></ul>
  21. 22. <ul><li>Questions recoded so that </li></ul><ul><li>correct answer = 1, incorrect answer = 0 </li></ul><ul><li>(multiple response questions earned multiple points) </li></ul><ul><li>Correct answers summed by EPAS focus </li></ul><ul><li>Higher scores = greater knowledge </li></ul><ul><li>Possible points by EPAS focus area </li></ul><ul><ul><ul><li>Conceptual framework = 35 </li></ul></ul></ul><ul><ul><ul><li>Evidence-based knowledge = 26 </li></ul></ul></ul><ul><ul><ul><li>Diversity = 5 </li></ul></ul></ul><ul><ul><ul><ul><ul><li>TOTAL = 66 </li></ul></ul></ul></ul></ul>
  22. 23. <ul><li>Mann Whitney non parametric for independent groups to determine differences between the two face to face classes. </li></ul><ul><ul><li>No differences; two face to face classes combined for further analysis </li></ul></ul><ul><li>Mann Whitney non parametric to determine differences between the online class and the face to face class(es) </li></ul>
  23. 24. <ul><li>Students in online class scored higher in conceptual knowledge </li></ul><ul><li>(p = .003) and in overall knowledge (p = .006) </li></ul><ul><li>No statistically significant difference in evidence-based knowledge (p =.095 ) </li></ul><ul><li>or diversity knowledge ( = .202) </li></ul>
  24. 25. <ul><li>Differences in learning outcomes related to: </li></ul><ul><ul><li>Part-time/full time status NO </li></ul></ul><ul><ul><li>Undergraduate Major NO </li></ul></ul><ul><ul><li>Prior Online NO </li></ul></ul><ul><ul><li>Student Age NO </li></ul></ul>
  25. 26. <ul><li>Questions about: </li></ul><ul><ul><li>Professor behaviors and attitudes </li></ul></ul><ul><ul><li>Student general learning experience in the course </li></ul></ul><ul><ul><li>Student knowledge and skill acquisition related to course objectives </li></ul></ul>
  26. 27. <ul><li>Differences in learning outcomes related to: </li></ul><ul><ul><li>Part-time/full time status NO </li></ul></ul><ul><ul><li>Undergraduate Major NO </li></ul></ul><ul><ul><li>Prior Online NO </li></ul></ul><ul><ul><li>Student Age NO </li></ul></ul>
  27. 29. <ul><li>Questions about: </li></ul><ul><li>Professor behaviors and attitudes </li></ul><ul><li>Student general learning experience in the course </li></ul><ul><li>Student knowledge and skill acquisition related to course objectives </li></ul>
  28. 30. <ul><li>Items included in Professor Behaviors </li></ul><ul><li>(4 = Strongly agree; 3 =Agree; 2= Disagree; 1=Strongly disagree) </li></ul><ul><li>The instructor ….. </li></ul><ul><li>created an environment of trust and fairness. </li></ul><ul><li>encouraged discussion, participation, and questions. </li></ul><ul><li>treated students with courtesy and respect </li></ul><ul><li>demonstrated sensitivity to my needs as a student </li></ul><ul><li>clearly defined the requirements for my work in the course </li></ul><ul><li>presented course material in an organized and informative manner </li></ul><ul><li>provided helpful feedback on papers, tests, presentations, or other assignments </li></ul><ul><li>addressed social and economic justice as related to the course </li></ul><ul><li>addressed diversity and oppressed populations as related to the course </li></ul>PROFESSOR BEHAVIORS AND ATTITUDES MEAN Total 3.73 Face to Face 3.744 Online 3.722
  29. 31. <ul><li>As a student in this course I ….. </li></ul><ul><li>(4 = Strongly agree; 3 =Agree; 2= Disagree; 1=Strongly disagree) </li></ul><ul><li>engaged in critical thinking and analysis </li></ul><ul><li>was motivated to learn about the course subject </li></ul><ul><li>developed greater self-awareness </li></ul><ul><li>developed significant skills as a result of the course </li></ul><ul><li>increased my knowledge of the subject matter of the course </li></ul><ul><li>increased my knowledge of social and economic justice as related to the course </li></ul><ul><li>increased my knowledge of diversity and oppressed populations as related to the course </li></ul><ul><li>Prior to taking this course I was interested in the course content </li></ul>STUDENT LEARNING EXPERIENCES IN THE COURSE MEAN Total 3.65 Face to Face 3.62 Online 3.73
  30. 32. <ul><li>To what degree did this course contribute to your ability to… </li></ul><ul><li>( 5=A great deal; 4=A lot; 3=Somewhat; 2=A little; 1=Not at all) </li></ul><ul><li>Practice personal reflection and self-correction to assure continual professional development </li></ul><ul><li>Recognize and manage personal values in a way that allows professional values to guide practice </li></ul><ul><li>Make ethical decisions by applying standards of the National Association of Social Workers Code of Ethics and, as applicable, of the International Federation of Social Workers/International Association of Schools of Social Work Ethics in Social Work, Statement of Principles </li></ul><ul><li>Tolerate ambiguity in resolving ethical conflicts </li></ul><ul><li>Apply strategies of ethical reasoning to arrive at principled decisions </li></ul><ul><li>Distinguish, appraise, and integrate multiple sources of knowledge, including research-based knowledge, and practice wisdom </li></ul><ul><li>Recognize the extent to which a culture’s structures and values may oppress, marginalize, alienate, or create or enhance privilege and power </li></ul><ul><li>Understand the forms and mechanisms of oppression and discrimination </li></ul><ul><li>Use research evidence to inform practice </li></ul><ul><li>Utilize conceptual frameworks to guide the processes of assessment, intervention, and evaluation </li></ul><ul><li>Critique and apply knowledge to understand person and environment </li></ul><ul><li>Substantively and affectively prepare for action with individuals, families, groups, organizations, and communities </li></ul><ul><li>Use empathy and other interpersonal skills </li></ul><ul><li>Gain sufficient self-awareness to eliminate the influence of personal biases and values in working with diverse groups </li></ul><ul><li>Recognize and communicate their understanding of the importance of difference in shaping life experiences </li></ul>STUDENT KNOWLEDGE AND SKILL ACQUISITION RELATED TO COURSE OBJECTIVES MEAN Total 4.36 Face to Face 4.2920 Online 4.5147
  31. 33. Student time
  32. 34. <ul><li>More part-time students self-select into Online Course </li></ul><ul><li>Greater Conceptual and Overall Learning Outcomes in Online Course </li></ul><ul><li>More expenditure of faculty time in Online Course </li></ul><ul><ul><li>Greatest difference in grading Blackboard CLG’s </li></ul></ul><ul><li>Similar use of technology in both courses </li></ul><ul><li>Student perception of learning slightly higher in online course </li></ul><ul><li>Student perception of instructor slightly higher in face to face course </li></ul><ul><li>Student perception of workload demands greater in online course </li></ul>
  33. 35. <ul><li>Evaluations of overall social work programs as opposed to courses </li></ul><ul><li>What happens to field </li></ul><ul><li>Social work educators inclination, knowledge, skills regarding online education </li></ul>
  34. 36. <ul><ul><ul><li>Allen, I.E., & Seaman, J. (2010). Learning on demand: Online education in the United States, 2009. Babson Survey Research Group. </li></ul></ul></ul><ul><ul><ul><li>Bernard, R.M., et al. (2004). How Does Distance Education Compare With Classroom Instruction? A Meta-Analysis of the Empirical Literature? Review of Educational Research, 74 (3), 379-439. </li></ul></ul></ul><ul><li>Coe, J., & Elliott, D. (1999). An evaluation of teaching direct practice courses in a distance education program for rural settings. Journal of Social Work Education, 35 (3), 353-365. </li></ul><ul><li>Coe Regan, J.R., Freddolino, P., Schoech, D., & Watkins, J. (2007). Utilizing evidence-based research in developing best practice guidelines for distance education in social work. Paper presentation at the Council on Social Work Education Annual Program Meeting (CSWE-APM) in San Francisco, CA on October 30, 2007. </li></ul><ul><li>Coe Regan, J.R., & Youn, E.J. (2008). Past, present, and future trends in teaching clinical skills through web-based learning environments. Journal of Social Work Education, 44 (2), 95-115. </li></ul><ul><li>Faux, T., & Black-Hughes, C. (2000). A comparison of using the internet versus lectures to teach social work history. Research on Social Work Practice, 10 (4), 454-466. </li></ul><ul><li>Hutchison, E. (2008). Dimensions of human behavior: The changing life course (3rd ed.) . Thousand Oaks, CA: Pine Forge Press. </li></ul>
  35. 37. <ul><li>Macy, J.A., Rooney, R.H., Hollister, C.D., & Freddolino, P.P. (2001). Evaluation of distance education programs in social work. Journal of Technology in Human Services, 18 (3/4), 63-84. </li></ul><ul><li>Newhouse, J.J. (2001). Successful distance learning graduate education in human services. Journal of Technology in Human Services, 18 (1/2), 33-50. </li></ul><ul><li>Packard, T. & Austin, M. (2009). Using a Comprehensive Case-Based Examination To Evaluate and Integrate Student Learning in Social Work Administration', Journal of Teaching in Social Work, 29: 2, 204 - 215. </li></ul><ul><ul><ul><li>Seabury, B.A. (2005). An evaluation of on-line, interactive tutorials designed to teach practice concepts. Journal of Teaching in Social Work, 25 (1), 103-115. </li></ul></ul></ul><ul><li>Seaman, J. (2009). Online Learning as a Strategic Asset. Volume II: The Paradox of Faculty Voices: Views and Experiences with Online Learning Results of a National Faculty Survey, Part of the Online Education Benchmarking Study Co ducted by the A♦P♦L♦U-Sloan National Commission on Online Learning. Association of Public and Land-grant Universities: Washington, D.C. </li></ul><ul><ul><ul><li>Siebert, D.C., Siebert, C.F., & Spaulding-Givens, J. (2006). Teaching clinical social work skills primarily online: An evaluation. Journal of Social Work Education, 42 (2), 325-336. </li></ul></ul></ul><ul><li>Siegel, E., Jennings, J. G., Conklin, J., & Flynn, S.A.N. (1998). Distance learning in social work education: results and implications of a national survey. Journal of Social Work Education, 34 (1), 71-80. </li></ul>
  36. 38. <ul><li>Thyer, B., Artlet, T., Markward, M., & Dozier, C. (1998). Evaluating distance learning in social work education: A replication study. Journal of Social Work Education, 34 (2), 291-296. </li></ul><ul><li>U. S. Department of Education, Office of Planning, Evaluation, and Policy Development. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies [Electronic Version] . Retrieved from, </li></ul><ul><li>Vernon, R., Vakalahi, H., Pierce, D., Pittman-Munke, P., & Adkins, L. F. (2009). Distance education programs in social work: current and emerging trends. Journal of Social Work Education, 45 (2), 263-276. </li></ul><ul><li>Wilke, D., & Vinton, L. (2006). Evaluation of the first web-based advanced standing msw program. Journal of Social Work Education, 42 (3), 607-620. </li></ul><ul><ul><ul><li>Woehle, R., & Quinn, A. (2009). An experiment comparing HBSE graduate social work classes: Face-to-face and at a distance. Journal of Teaching in Social Work, 29 (4), 418-430. </li></ul></ul></ul><ul><li>York, R. (2008). Comparing three modes of instruction in a graduate social work program. Journal of Social Work Education, 44 (2), 157-172. </li></ul>