Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CypherWorx OST Effiacy Study Results 2015

103 views

Published on

  • Be the first to comment

  • Be the first to like this

CypherWorx OST Effiacy Study Results 2015

  1. 1. San Antonio Pilot Study: Data Analysis Results Summary of Data Analysis Findings July, 2015 1
  2. 2. Background This Efficacy Study was conducted with Excel Beyond the Bell-San Antonio. Excel Beyond the Bell- San Antonio (EBBSA) is a collaborative effort among nonprofit and public agencies who provide Out- of-School Time (OST) or related services to youth and their families. EBBSA works with these and other willing partners to seek a shared vision for San Antonio youth. The purpose of this study was to assess the effectiveness of six e-learning courses. At least two participants at each school-age child care provider member agency served by the EBBSA Collaborative were asked to participate in this efficacy study of six e-learning courses. The participants were chosen to represent a wide range of experience and organization size: they included new hires, seasoned professionals, those with high school/GED education, those with masters or doctoral degrees, front-line staff, and directors. A total of 147 people completed the six e-learning courses, and pre- and post-surveys. Dr. James Marshall, an expert in e-learning and faculty member at San Diego State University, received the full dataset collected by CypherWorx. He then conducted a topline analysis of participant test performance and survey item responses. This document summarizes Dr. Marshall’s analysis of the provided dataset. Pilot Study Analysis Findings 2
  3. 3. About Dr. Marshall James Marshall Ph.D. is a faculty member in the Department of Learning Design and Technology (formerly Educational Technology) at San Diego State University. With over 20 years in the learning design field, he brings a wealth of applied experience to his teaching and research endeavors. An expert in e-learning and design-based research, Dr. Marshall’s research focuses on the relationship between an e-learning product’s design, and the results it achieves. Typically, this involves examining immediate knowledge gains for learners, and ultimately, return on investment for the organizations in which these learners work. Jim has been responsible for evaluating over $25 million dollars worth of federal-, state- and locally-funded programs for purposes of optimizing their design, and quantifying their impact. Through his consulting work, he has collaborated with organizations that include: Bank of America, Anheuser Busch, Court TV, McGraw Hill Companies, The Princeton Review, The Transportation Security Administration, TIAA-CREF, The Corporation for Public Broadcasting and the U.S. Department of Education. His most recent research can be found in the Journal of Applied Instructional Design, Performance Improvement Quarterly and the International Journal of E-Learning. Education B.A. in Liberal Studies, San Diego State University M.A. in Educational Technology, San Diego State University Ph.D. in Education, Claremont Graduate University and San Diego State University Preliminary Findings 3
  4. 4. List of Courses Tested Course Title 3 Exploring Developmental Needs and Characteristics of Different Age Groups (Implications for Programming) 6 Guiding the Behavior of Individual Children 8 Human Relations Skill Development 20 Developing Activities that Encourage Creativity and Cognitive Development 24 Helping Children with ADD Succeed in School-Age Programs 28 Commitment to Quality in School-Age Programs 4Pilot Study Analysis Findings Data was collected for six of the e-learning courses. The involved data included: • Pre- and posttest scores—based on answers to test items aligned with each course’s objectives. The Objective Measures section of this document summarizes test results. • Pre- and postsurvey responses—from a survey conducted with participants prior to engaging in e-learning and following completion of the final course. The Participant Beliefs section of this document summarizes survey results. EBBSA selected the following e-learning courses for this study.
  5. 5. Course Content State Approvals 5Pilot Study Analysis Findings The course content chosen for the study has been approved in Texas, and the States in Green as of 7/8/15.
  6. 6. PARTICIPANT DEMOGRAPHICS Information describing the 147 pilot study participants 6Pilot Study Analysis Findings
  7. 7. Demographics: Gender Pilot Study Analysis Findings 7 Male 24% Female 76% Study Participants (n=147) National Averages For Field* *Source: Understanding the Afterschool Workforce: Opportunities and Challenges for an Emerging Profession (NAA, 2006) Male 30% Female 70%
  8. 8. Demographics: Age Pilot Study Analysis Findings 8 18-29 years old 61% 30-39 years old 15% 40-49 years old 12% 50-59 years old 8% 60+ years old 4% Study Participants (n=147) National Averages For Field* *Source: Understanding the Afterschool Workforce: Opportunities and Challenges for an Emerging Profession (NAA, 2006) 18-29 years old 47% 30-39 years old 21% 40-49 years old 17% 50-59 years old 10% 60+ years old 5%
  9. 9. 2-year degree 11% 4-year degree 28% Advanced Degree (Masters or Doctoral) 17% GED 1% High School Diploma 42% Other 1% Demographics: Education 9Pilot Study Analysis Findings 2-year degree 8% 4-year degree 37% Advanced Degree (Masters or Doctoral) 15% GED 1% High School Diploma 37% Other 2% Study Participants (n=147) National Averages For Field* *Source: Understanding the Afterschool Workforce: Opportunities and Challenges for an Emerging Profession (NAA, 2006)
  10. 10. Demographics: Professional Experience 10 0 - 1 years 18% 2 - 5 years 43% 6 - 10 years 18% 11 - 15 years 8% 16+ years 13% Pilot Study Analysis Findings Study Participants (n=147) National Averages For Field* *Source: Understanding the Afterschool Workforce: Opportunities and Challenges for an Emerging Profession (NAA, 2006) 0 - 1 years 16% 2 - 5 years 41% 6 - 10 years 19% 11 - 15 years 9% 16+ years 15%
  11. 11. Demographics: Study Participant Position Title Position Title Number of Participants Percentage of Participants Admin 2 1.4% Aide / Assistant Teacher 50 34.0% Assistant Director / Coordinator 5 3.4% Clinician 1 .7% Director/Owner 26 17.7% Education Coordinator 4 2.7% Facilitator 2 1.4% Juvenile Probation Counselor 1 .7% Lead Teacher/Supervisor 15 10.2% Site Manager 28 19.0% Youth Worker 13 8.8% Totals 147 100% 11Pilot Study Analysis Findings
  12. 12. Demographics: Computer Skills 12 Novice 1% Low Level 3%Intermediate 38% High Level 53% Very Skilled 5% Study Participant Self-rating: How would you rate your computer knowledge and competency? (n=147) Pilot Study Analysis Findings
  13. 13. OBJECTIVE MEASURES A comparison of pre- and posttest performance to determine growth as a result of instruction 13Pilot Study Analysis Findings
  14. 14. Pretest to Posttest Comparisons Each course in this pilot includes a pre- and posttest instrument. Aligned with the course objectives, these tests were used to assess participant knowledge prior to, and following completion of, each course. This analysis uses mean scores to analyze differences in pre- and posttest performance. In addition, we examine whether found differences are (a) statistically significant— meaning unlikely to be the result of random chance; and (b) consistent for all participants, regardless of various demographics that could influence performance (i.e., age or years in the profession). Pilot Study Analysis Findings 14
  15. 15. Pretest to Posttest Gains 56.1 SD=21.0 70.2 SD=19.4 55.0 SD=24.0 66.0 SD=23.7 60.6 SD=23.4 53.2 SD=23.8 83.9 SD=14.7 87.7 SD=7.6 87.7 SD=8.9 89.8 SD=10.8 85.6 SD=10.6 83.2 SD=12.1 40 50 60 70 80 90 100 Exploring Developmental Needs Guiding Behavior of Individual Children Human Relation Skill Development Activities for Creativity and Cognitive Development Helping Children with ADD Succeed Commitment to Quality in School- Age Programs PercentofItemsAnsweredCorrectly Pretest (n=147) Posttest (n=147) 15 +27.8 +17.5 +32.7 +30.0 Statistically Significant Statistically Significant Statistically Significant Statistically Significant Statistically Significant Statistically Significant +25.0+23.8 Pilot Study Analysis Findings
  16. 16. Pretest to Posttest Gains On average, participants increased their knowledge in each of the six courses, with statistically significant differences. • Significance levels for pre-to-posttest differences on all six tests were p=.000. • This finding indicates that the observed differences between pre- and posttest mean scores had essentially no possibility of occurring by random chance. • We conclude the difference (growth) is attributable to the intervention (in this case, the training provided to participants). 16 56.1 SD=21.0 70.2 SD=19.4 55.0 SD=24.0 66.0 SD=23.7 60.6 SD=23.4 53.2 SD=23.8 83.9 SD=14.7 87.7 SD=7.6 87.7 SD=8.9 89.8 SD=10.8 85.6 SD=10.6 83.2 SD=12.1 40 50 60 70 80 90 100 Exploring Developmental Needs Guiding Behavior of Individual Children Human Relation Skill Development Activities for Creativity and Cognitive Development Helping Children with ADD Succeed Commitment to Quality in School- Age Programs PercentofItemsAnsweredCorrectly Pretest Posttest +27.8 +17.5 +32.7 +30.0 Statistically Significant Statistically Significant Statistically Significant Statistically Significant Statistically Significant Statistically Significant +25.0+23.8 Pilot Study Analysis Findings
  17. 17. Pretest to Posttest Gains Course (n=147) Pretest: Mean Score Pretest: Standard Deviation Posttest: Mean Score Posttest: Standard Deviation sig Course 3 56.1 21.0 83.9 14.7 .000 Course 6 70.2 19.4 87.7 7.6 .000 Course 8 55.0 24.0 87.7 8.9 .000 Course 20 66.0 23.7 89.8 10.8 .000 Course 24 60.6 23.4 85.6 10.6 .000 Course 28 53.2 23.8 83.2 12.1 .000 • Pretest score distributions varied more (as indicated by the standard deviation), relative to posttest distributions. For example, the standard deviation (variance) on Course 3 shifted from 21.0 points on the pretest to 14.7 on the posttest— representing an almost 1/3 reduction in response variance). • This means that while pretest scores varied greatly, posttest scores were clustered closer to the mean. • Regardless of where an individual pretested, most participants performed consistently higher on the posttest—with far less variation in scores. The following slides illustrate these changes in distribution for each course. 17Pilot Study Analysis Findings
  18. 18. Change in Distribution: Course 3 Pretest: Mean = 56.1 | SD = 21.0 Posttest: Mean = 83.9 | SD = 14.7 18 NumberofTestswithScore NumberofTestswithScore A comparison of the two distributions illustrates how test scores shifted to higher points on the scale from pretest to posttest. Additionally, the variance in scores was greatly reduced—with most posttest scores clustering around the 87.7 mean score. Pilot Study Analysis Findings Exploring Developmental Needs and Characteristics of Different Age Groups: Implications for Programming
  19. 19. Change in Distribution: Course 6 Pretest: Mean = 70.2 | SD = 19.4 Posttest: Mean = 87.7 | SD = 7.6 19 NumberofTestswithScore NumberofTestswithScore A comparison of the two distributions illustrates how test scores shifted to higher points on the scale from pretest to posttest. Additionally, the variance in scores was greatly reduced—with most posttest scores clustering around the 87.7 mean score. Pilot Study Analysis Findings Guiding the Behavior of Individual Children
  20. 20. Change in Distribution: Course 8 Pretest: Mean = 55.0 | SD = 24.0 Posttest: Mean = 87.7 | SD = 8.9 20 NumberofTestswithScore NumberofTestswithScore A comparison of the two distributions illustrates how test scores shifted to higher points on the scale from pretest to posttest. Additionally, the variance in scores was greatly reduced—with most posttest scores clustering around the 87.7 mean score. Pilot Study Analysis Findings Human Relations Skill Development
  21. 21. Change in Distribution: Course 20 Pretest: Mean = 66.0 | SD = 23.7 Posttest: Mean = 89.8 | SD = 10.8 21 NumberofTestswithScore NumberofTestswithScore A comparison of the two distributions illustrates how test scores shifted to higher points on the scale from pretest to posttest. Additionally, the variance in scores was greatly reduced—with most posttest scores clustering around the 87.7 mean score. Pilot Study Analysis Findings Developing Activities That Encourage Creativity and Cognitive Development
  22. 22. Change in Distribution: Course 24 Pretest: Mean = 60.6 | SD = 23.4 Posttest: Mean = 85.6 | SD = 10.6 22 NumberofTestswithScore NumberofTestswithScore A comparison of the two distributions illustrates how test scores shifted to higher points on the scale from pretest to posttest. Additionally, the variance in scores was greatly reduced—with most posttest scores clustering around the 87.7 mean score. Pilot Study Analysis Findings Helping Children with ADD Succeed in School-Age Programs
  23. 23. Change in Distribution: Course 28 Pretest: Mean = 53.2 | SD = 23.8 Posttest: Mean = 83.2 | SD = 12.1 23 NumberofTestswithScore NumberofTestswithScore A comparison of the two distributions illustrates how test scores shifted to higher points on the scale from pretest to posttest. Additionally, the variance in scores was greatly reduced—with most posttest scores clustering around the 87.7 mean score. Pilot Study Analysis Findings Commitment to Quality in School-Age Programs
  24. 24. Pretest to Posttest Gains While performance on the pretest often varied based on key demographics, posttest scores proved consistent regardless of potential demographically-based advantages and/or disadvantages. The average level of performance on almost all posttests was determined to be consistent, regardless of the participants’: • age • gender • level of education • years in the profession • computer skills 24 56.1 SD=21.0 70.2 SD=19.4 55.0 SD=24.0 66.0 SD=23.7 60.6 SD=23.4 53.2 SD=23.8 83.9 SD=14.7 87.7 SD=7.6 87.7 SD=8.9 89.8 SD=10.8 85.6 SD=10.6 83.2 SD=12.1 40 50 60 70 80 90 100 Exploring Developmental Needs Guiding Behavior of Individual Children Human Relation Skill Development Activities for Creativity and Cognitive Development Helping Children with ADD Succeed Commitment to Quality in School- Age Programs PercentofItemsAnsweredCorrectly Pretest Posttest +27.8 +17.5 +32.7 +30.0 Statistically Significant Statistically Significant Statistically Significant Statistically Significant Statistically Significant Statistically Significant +25.0+23.8 Pilot Study Analysis Findings
  25. 25. Pretest to Posttest Gains: Gender 49.1 61.0 46.1 60.1 51.1 42.4 82.6 86.7 89.3 87.6 85.6 84.0 58.3 73.1 57.8 67.8 63.6 56.6 84.3 88.0 87.2 90.4 85.6 83.0 40 50 60 70 80 90 100 Exploring Developmental Needs Guiding Behavior of Individual Children Human Relation Skill Development Activities for Creativity and Cognitive Development Helping Children with ADD Succeed Commitment to Quality in School- Age Programs PercentofItemsAnsweredCorrectly Male Pretest (n=35) Male Posttest (n=35) Female Pretest (n=112) Female Posttest (n=112) 25Pilot Study Analysis Findings • All pre- to posttest differences are statistically significant. • Posttest scores did not differ significantly between male and female participants.
  26. 26. Pretest to Posttest Gains: Age 56.9 68.3 52.9 82.8 87.2 87.9 48.2 70.2 54.1 83.6 86.3 85.9 44.4 66.7 52.8 85.5 87.2 88.3 77.3 87.7 69.1 89.5 92.7 87.3 70.0 77.5 70.0 85.0 91.7 90.8 40 50 60 70 80 90 100 Exploring Developmental Needs Guiding Behavior of Individual Children Human Relation Skill Development PercentofItemsAnsweredCorrectly 18-29 Pretest (n=90) 18-29 Posttest (n=90) 30-39 Pretest (n=22) 30-39 Posttest (n=22) 40-49 Pretest (n=18) 40-49 Posttest (n=18) 50-59 Pretest (n=11) 50-59 Posttest (n=11) 60+ Pretest (n=6) 60+ Posttest (n=6) 26Pilot Study Analysis Findings • Pre- to posttest differences for ages 18-29 through 40-49 are statistically significant. Small sample sizes for ages 50-59 and 60+ limit the ability to analyze for these groups. • Posttest scores did not differ significantly by age for these three courses.
  27. 27. Pretest to Posttest Gains: Age 65.2 57.2 51.7 89.2 85.7 82.7 62.0 58.2 51.6 90.0 81.6 80.0 59.4 63.1 50.8 88.9 85.5 90.8 79.1 80.4 68.2 93.2 90.4 85.0 86.7 76.7 62.5 93.3 90.0 76.7 40 50 60 70 80 90 100 Activities for Creativity and Cognitive Development Helping Children with ADD Succeed Commitment to Quality in School- Age Programs PercentofItemsAnsweredCorrectly 18-29 Pretest (n=90) 18-29 Posttest (n=90) 30-39 Pretest (n=22) 30-39 Posttest (n=22) 40-49 Pretest (n=18) 40-49 Posttest (n=18) 50-59 Pretest (n=11) 50-59 Posttest (n=11) 60+ Pretest (n=6) 60+ Posttest (n=6) 27Pilot Study Analysis Findings • Pre- to posttest differences for ages 18-29 through 40-49 are statistically significant. Small sample sizes for ages 50-59 and 60+ limit the ability to analyze for these groups. • Posttest scores differed significantly between 30-39 and 40-49 year old participants for the final course displayed. All other posttest scores did not differ significantly based on participant age.
  28. 28. Pretest to Posttest Gains: Education Level 59.6 73.1 55.2 84.0 89.2 87.3 54.1 65.0 53.8 80.9 87.2 90.9 54.0 66.6 51.0 85.1 86.5 87.3 51.6 72.4 62.2 83.4 86.4 87.8 40 50 60 70 80 90 100 Exploring Developmental Needs Guiding Behavior of Individual Children Human Relation Skill Development PercentofItemsAnsweredCorrectly High School Diploma Pretest (n=62) High School Diploma Posttest (n=62) 2-year Degree Pretest (n=16) 2-year Degree Posttest (n=16) 4-year degree Pretest (n=41) 4-year Degree Posttest (n=41) Advanced Degree Pretest (n=25) Advanced Degree Posttest (n=25) 28Pilot Study Analysis Findings • Pre- to posttest differences for were statistically significant for each education level-based group. • Posttest scores did not differ significantly based on education level.
  29. 29. Pretest to Posttest Gains: Education Level 69.6 61.8 55.7 87.8 85.1 82.1 60.6 59.1 55.3 91.3 85.6 84.4 60.9 55.7 47.8 92.6 87.1 85.4 67.8 64.2 53.4 89.2 84.6 81.2 40 50 60 70 80 90 100 Activities for Creativity and Cognitive Development Helping Children with ADD Succeed Commitment to Quality in School- Age Programs PercentofItemsAnsweredCorrectly High School Diploma Pretest (n=62) High School Diploma Posttest (n=62) 2-year Degree Pretest (n=16) 2-year Degree Posttest (n=16) 4-year degree Pretest (n=41) 4-year Degree Posttest (n=41) Advanced Degree Pretest (n=25) Advanced Degree Posttest (n=25) 29Pilot Study Analysis Findings • Pre- to posttest differences for were statistically significant for each education level-based group. • Posttest scores did not differ significantly based on education level.
  30. 30. Pretest to Posttest Gains: Professional Experience 55.2 71.7 50.7 85.4 87.6 87.6 56.7 68.2 52.4 80.1 87.2 87.4 55.5 71.1 57.0 86.8 87.4 87.2 53.6 69.1 55.0 82.7 86.4 88.2 57.6 74.2 66.8 90.5 90.5 89.5 40 50 60 70 80 90 100 Exploring Developmental Needs Guiding Behavior of Individual Children Human Relation Skill Development PercentofItemsAnsweredCorrectly 0-1 Years Pretest (n=27) 0-1 Years Posttest (n=27) 2-5 Years Pretest (n=63) 2-5 Years Posttest (n=63) 6-10 Years Pretest (n=27) 6-10 Years Posttest (n=27) 11-15 Years Pretest (n=11) 11-15 Years Posttest (n=11) 16+ Years Pretest (n=19) 30Pilot Study Analysis Findings • Pre- to posttest differences for were statistically significant for each professional experience-based group. • Posttest scores did not differ significantly based on years of professional experience.
  31. 31. Pretest to Posttest Gains: Professional Experience 64.2 56.3 50.7 88.7 85.2 82.2 62.8 58.2 50.7 90.0 83.8 81.9 69.6 62.0 55.0 92.6 88.0 85.0 64.5 54.1 54.5 85.0 84.1 83.2 74.7 76.6 61.8 92.6 89.7 86.6 40 50 60 70 80 90 100 Activities for Creativity and Cognitive Development Helping Children with ADD Succeed Commitment to Quality in School- Age Programs PercentofItemsAnsweredCorrectly 0-1 Years Pretest (n=27) 0-1 Years Posttest (n=27) 2-5 Years Pretest (n=63) 2-5 Years Posttest (n=63) 6-10 Years Pretest (n=27) 6-10 Years Posttest (n=27) 11-15 Years Pretest (n=11) 11-15 Years Posttest (n=11) 16+ Years Pretest (n=19) 31Pilot Study Analysis Findings • Pre- to posttest differences for were statistically significant for each professional experience-based group. • Posttest scores did not differ significantly based on years of professional experience.
  32. 32. Pretest to Posttest Gains: Computer Skills 72.0 93.0 67.0 88.0 97.0 89.0 57.8 70.5 54.4 85.5 87.7 89.5 54.3 68.6 54.1 83.4 87.1 86.3 54.3 72.8 62.1 72.8 87.8 89.3 40 50 60 70 80 90 100 Exploring Developmental Needs Guiding Behavior of Individual Children Human Relation Skill Development PercentofItemsAnsweredCorrectly Low Pretest (n=5) Low Posttest (n=5) Intermediate Pretest (n=56) Intermediate Posttest (n=56) High Pretest (n=78) High Posttest (n=78) Very Skilled Pretest (n=7) Very Skilled Posttest (n=7) 32Pilot Study Analysis Findings • Pre- to posttest differences for intermediate and high computer skills groups are statistically significant. Small sample sizes for the low and very skilled groups limit the ability to analyze for these groups. • Posttest scores did not differ significantly based on computer skills between intermediate and high computer skills groups. Small sample sizes for the low and very skilled groups limit the ability to analyze for these groups.
  33. 33. Pretest to Posttest Gains: Computer Skills 77.0 80.0 71.0 95.0 89.0 84.0 65.7 62.0 57.8 90.8 87.8 85.4 65.2 58.8 48.9 88.5 83.7 82.0 66.4 55.7 55.7 91.4 87.8 85.7 40 50 60 70 80 90 100 Activities for Creativity and Cognitive Development Helping Children with ADD Succeed Commitment to Quality in School- Age Programs PercentofItemsAnsweredCorrectly Low Pretest (n=5) Low Posttest (n=5) Intermediate Pretest (n=56) Intermediate Posttest (n=56) High Pretest (n=78) High Posttest (n=78) Very Skilled Pretest (n=7) Very Skilled Posttest (n=7) 33Pilot Study Analysis Findings • Pre- to posttest differences for intermediate and high computer skills groups are statistically significant. Small sample sizes for the low and very skilled groups limit the ability to analyze for these groups. • Posttest scores did not differ significantly based on computer skills between intermediate and high computer skills groups. Small sample sizes for the low and very skilled groups limit the ability to analyze for these groups.
  34. 34. Pretest to Posttest Gains: Course 3, by Position 59.7 40.0 57.7 61.3 52.0 56.1 55.0 83.2 82.0 86.7 86.3 86.0 84.8 79.2 30 40 50 60 70 80 90 100 Aide / Assistant Teacher (n=50) Assistant Director / Coordinator (n=5) Director/Owner (n=26) Education Coordinator (n=4) Lead Teacher/Supervisor (n=15) Site Manager (n=28) Youth Worker (n=13) PercentofItemsAnsweredCorrectly Pretest Posttest 34Pilot Study Analysis Findings • Pre- to posttest differences for Aide/Assistant Teacher, Director/Owner, Lead Teacher/Supervisor and Site Manager are statistically significant. Small sample sizes limit the ability to analyze for the remaining groups. • Posttest scores did not differ significantly based on position for the groups referenced by name in the preceding bullet. Small sample sizes limit the ability to analyze for the remaining groups. Exploring Developmental Needs and Characteristics of Different Age Groups: Implications for Programming
  35. 35. Pretest to Posttest Gains: Course 6, by Position 73.8 59.0 71.2 75.0 70.0 71.1 60.4 88.9 82.0 89.4 82.5 87.7 86.1 86.5 40 50 60 70 80 90 100 Aide / Assistant Teacher (n=50) Assistant Director / Coordinator (n=5) Director/Owner (n=26) Education Coordinator (n=4) Lead Teacher/Supervisor (n=15) Site Manager (n=28) Youth Worker (n=13) PercentofItemsAnsweredCorrectly Pretest Posttest 35Pilot Study Analysis Findings • Pre- to posttest differences for Aide/Assistant Teacher, Director/Owner, Lead Teacher/Supervisor and Site Manager are statistically significant. Small sample sizes limit the ability to analyze for the remaining groups. • Posttest scores did not differ significantly based on position for the groups referenced by name in the preceding bullet. Small sample sizes limit the ability to analyze for the remaining groups. Guiding the Behavior of Individual Children
  36. 36. Pretest to Posttest Gains: Course 8, by Position 56.9 57.0 58.8 57.5 58.0 50.7 49.6 88.7 86.0 88.7 82.5 86.7 85.7 90.4 40 50 60 70 80 90 100 Aide / Assistant Teacher (n=50) Assistant Director / Coordinator (n=5) Director/Owner (n=26) Education Coordinator (n=4) Lead Teacher/Supervisor (n=15) Site Manager (n=28) Youth Worker (n=13) PercentofItemsAnsweredCorrectly Pretest Posttest 36Pilot Study Analysis Findings • Pre- to posttest differences for Aide/Assistant Teacher, Director/Owner, Lead Teacher/Supervisor and Site Manager are statistically significant. Small sample sizes limit the ability to analyze for the remaining groups. • Posttest scores did not differ significantly based on position for the groups referenced by name in the preceding bullet. Small sample sizes limit the ability to analyze for the remaining groups. Human Relations Skill Development
  37. 37. Pretest to Posttest Gains: Course 20, by Position 75.2 64.0 66.3 57.5 62.3 61.1 56.5 88.9 83.0 91.3 90.0 89.7 90.7 90.8 40 50 60 70 80 90 100 Aide / Assistant Teacher (n=50) Assistant Director / Coordinator (n=5) Director/Owner (n=26) Education Coordinator (n=4) Lead Teacher/Supervisor (n=15) Site Manager (n=28) Youth Worker (n=13) PercentofItemsAnsweredCorrectly Pretest Posttest 37Pilot Study Analysis Findings • Pre- to posttest differences for Aide/Assistant Teacher, Director/Owner, Lead Teacher/Supervisor and Site Manager are statistically significant. Small sample sizes limit the ability to analyze for the remaining groups. • Posttest scores did not differ significantly based on position for the groups referenced by name in the preceding bullet. Small sample sizes limit the ability to analyze for the remaining groups. Developing Activities That Encourage Creativity and Cognitive Development
  38. 38. Pretest to Posttest Gains: Course 24, by Position 61.9 62.0 64.8 58.8 63.3 60.2 48.5 85.1 80.0 88.1 80.0 84.3 86.3 88.1 40 50 60 70 80 90 100 Aide / Assistant Teacher (n=50) Assistant Director / Coordinator (n=5) Director/Owner (n=26) Education Coordinator (n=4) Lead Teacher/Supervisor (n=15) Site Manager (n=28) Youth Worker (n=13) PercentofItemsAnsweredCorrectly Pretest Posttest 38Pilot Study Analysis Findings • Pre- to posttest differences for Aide/Assistant Teacher, Director/Owner, Lead Teacher/Supervisor and Site Manager are statistically significant. Small sample sizes limit the ability to analyze for the remaining groups. • Posttest scores did not differ significantly based on position for the groups referenced by name in the preceding bullet. Small sample sizes limit the ability to analyze for the remaining groups. Helping Children with ADD Succeed in School-Age Programs
  39. 39. Pretest to Posttest Gains: Course 28, by Position 57.7 54.0 57.3 33.8 46.7 53.0 46.9 81.9 74.0 87.3 76.3 80.3 83.9 86.9 30 40 50 60 70 80 90 100 Aide / Assistant Teacher (n=50) Assistant Director / Coordinator (n=5) Director/Owner (n=26) Education Coordinator (n=4) Lead Teacher/Supervisor (n=15) Site Manager (n=28) Youth Worker (n=13) PercentofItemsAnsweredCorrectly Pretest Posttest 39Pilot Study Analysis Findings • Pre- to posttest differences for Aide/Assistant Teacher, Director/Owner, Lead Teacher/Supervisor and Site Manager are statistically significant. Small sample sizes limit the ability to analyze for the remaining groups. • Posttest scores did not differ significantly based on position for the groups referenced by name in the preceding bullet. Small sample sizes limit the ability to analyze for the remaining groups. Commitment to Quality in School-Age Programs
  40. 40. PARTICIPANT BELIEFS A comparison of participant self-assessment responses—prior to, and following, instruction 40Pilot Study Analysis Findings
  41. 41. Presurvey to Postsurvey Comparisons Project participants completed the same survey prior to their first course, and following their sixth course. The instrument challenged participants to self- assess their understanding of key concepts from each of the six courses. Participants were instructed as follows: On a Scale of 1-5 (with 1 being very little to no experience and 5 being high level of experience) would you rate your current knowledge of [key concept] Pilot Study Analysis Findings 41 1 2 3 4 5 Little or No Experience High Level of Experience
  42. 42. Survey Questions Our analysis used mean scores to understand pre-to-postsurvey differences. Mean scores represent the average score across the full group of participants on the provided rating scale. As an example, a mean score of 4.75 for a given item would indicate a considerable level of experience that approaches the “High Level of Experience” category (equal to 5). Pilot Study Analysis Findings 42 1 2 3 4 5 Little or No Experience High Level of Experience
  43. 43. Summary of Survey Results • All postsurvey ratings were higher relative to presurvey ratings, save a single item where the pre- and postsurvey means were identical. • For 22 of 27 items, the participant-indicated growth was analyzed to be statistically significant (Paired t-Test procedure), indicating little to no chance the observed difference resulted from random chance. • The highest levels of growth across a given course’s content was observed for two of the most applied courses—Developing Activities that Encourage Creativity and Cognitive Development, and Helping Children with ADD Succeed in School-Age Programs • As with the objective measures, postsurvey ratings did not differ significantly when participants were compared based on age, computer skills and education level. Pilot Study Analysis Findings 43
  44. 44. Course 3 44 3.5 3.6 3.5 3.0 3.2 3.4 1 2 3 4 5 Identifying and implementing strategies for addressing the seven developmental needs of early adolescence in your OST program The concept of “developmental diversity” and its implications for program planning in your OST program Developing ideas and implementing strategies for addressing the interests and needs of 5-12 year olds in your OST program Presurvey Postsurvey Little/No Experience High Level of Experience Statistically Significant Statistically Significant Cognitive Results (for reference) Pretest Posttest Gain Exploring Developmental Needs and Characteristics of Different Age Groups 56.1% 83.9% +27.8 On a Scale of 1-5, please rate your current knowledge of: Pilot Study Analysis Findings Exploring Developmental Needs and Characteristics of Different Age Groups: Implications for Programming
  45. 45. Course 6 45 3.9 3.9 3.8 3.6 3.6 3.8 3.5 3.4 1 2 3 4 5 Using positive guidance techniques to help children develop self-discipline and self-direction in your OST program Setting rules, limits, and consequences, and involving school- age children in the development of rules, limits, and consequences in your OST program Identifying common causes of conflicts and techniques for avoiding, minimizing, and managing conflicts when they do occur in your OST program Identifying and implementing strategies for guiding difficult behavior related to individual differences in temperament and individual problems with anger management in your OST program Presurvey Postsurvey Little/No Experience High Level of Experience Statistically Significant Statistically Significant Cognitive Results (for reference) Pretest Posttest Gain Guiding the Behavior of Individual Children 70.2% 87.7% +17.5 On a Scale of 1-5, please rate your current knowledge of: Pilot Study Analysis Findings Statistically Significant Guiding the Behavior of Individual Children
  46. 46. Course 8 Pilot Study Analysis Findings 46 3.7 3.7 3.8 3.6 3.9 3.8 3.7 3.5 3.5 3.8 3.5 3.6 3.6 3.5 1 2 3 4 5 Identifying components of the communication process, and the two types of listening: Passive Listening and Active Listening Identifying common barriers to communication in your OST program Identifying and implementing strategies for establishing positive relationships with children of all ages in your OST program Identifying and implementing strategies for stages of group development and how groups grow into teams in your OST program Identifying characteristics of effective teams, and factors that hurt or help team building in your OST program Identifying and implementing strategies to create a cooperative team atmosphere and solve problems with team members in your OST program Assessing personal human relations skills and setting priorities for improving human relations skills in your OST program Presurvey Postsurvey Little/No Experience High Level of Experience Statistically Significant Statistically Significant Cognitive Results (for reference) Pretest Posttest Gain Human Relations Skill Development 55.0% 87.7% +32.7 On a Scale of 1-5, please rate your current knowledge of: Statistically Significant Statistically Significant Human Relations Skill Development
  47. 47. Course 20 47 3.6 3.7 3.6 3.6 3.7 3.4 3.0 3.1 3.0 3.4 1 2 3 4 5 Understanding the three characteristics of creative people, and implementing techniques staff can use to create a program that fosters children’s creativity, curiosity, and sense of wonder Understanding creativity killers to avoid in your OST program Understanding and implementing the four components of the creative process and how to help children use this process as they plan and carry out their own activities and projects in your OST program Understanding the difference between soft and hard thinking and implementing the role of reasoning and other thinking skills in the creative process in your OST program Assessing and implementing strategies to use open-ended questions to help children develop their thinking skills, and utilizing the four types of open-ended questions that can be used to stimulate children’s logical thinking Presurvey Postsurvey Little/No Experience High Level of Experience Statistically Significant Statistically Significant Cognitive Results (for reference) Pretest Posttest Gain Developing Activities that Encourage Creativity and Cognitive Development 66.0% 89.8% +23.8 On a Scale of 1-5, please rate your current knowledge of: Pilot Study Analysis Findings Statistically Significant Statistically Significant Statistically Significant Developing Activities That Encourage Creativity and Cognitive Development
  48. 48. Course 24 48 Cognitive Results (for reference) Pretest Posttest Gain Helping Children with ADD Succeed in School-Age Programs 60.6% 85.6% +25.0 On a Scale of 1-5, please rate your current knowledge of: Pilot Study Analysis Findings 3.6 3.6 3.6 3.6 3.6 3.1 3.0 2.9 2.9 3.0 1 2 3 4 5 Identifying and discussing typical characteristics of children and youth with ADD Identifying how ADD can affect the development of school-age children and youth, and describe how typical characteristics and expectations of quality school-age programs can impact children and youth with ADD Identifying and implementing strategies for structuring and adapting school-age program environments and activities to accommodate children and youth with ADD, and helping children and youth structure their time and participation in your OST program Helping children and youth with ADD monitor and manage their own behavior, and establish positive relationships with others in your OST program Identifying and implementing strategies for being an effective advocate for children and youth with ADD in your OST program Presurvey Postsurvey Little/No Experience High Level of Experience Statistically Significant Statistically Significant Statistically Significant Statistically Significant Statistically Significant Helping Children with ADD Succeed in School-Age Programs
  49. 49. Course 28 Pilot Study Analysis Findings 49 3.5 3.3 3.6 3.2 2.4 2.8 1 2 3 4 5 Understanding key organizations, resources, and research initiatives that have contributed to the development of standards of quality in the field of school-age care Describing the School-Age Care Environment Rating Scale (SACERS) and how it can be used as a tool for assessing the quality of school-age programs/arranging levels of quality indicators for Items and Subscales of the SACERS in the appropriate order Understanding the purpose of the NAA Standards for Quality School-Age Care and the importance of making a commitment to continuous quality improvement Presurvey Postsurvey Little/No Experience High Level of Experience Statistically Significant Statistically Significant Cognitive Results (for reference) Pretest Posttest Gain Commitment to Quality in School-Age Programs 53.2% 83.2% +30.0 On a Scale of 1-5, please rate your current knowledge of: Statistically Significant Commitment to Quality in School-Age Programs

×