Program evaluation 2013

141 views
113 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
141
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Program evaluation 2013

  1. 1. Program Evaluation – Final Report Program Evaluation- Final Report Georgia Southern University FRIT 8435 Program Evaluation Lara Komanecky & Lisa Witteman Dr. Carlson Spring 2013
  2. 2. Program Evaluation – Final Report Table of Contents I. Executive Summary………………………………………………………………………………………………………………….…4 Purpose and Scope …………………………………………………………………………………………………………………….4 Audience…………………………………………………………………………………………………………………………………….4 Data Collection and Analysis………………………………………………………………………………………………………4 Findings…………………………………………………………………………………………………………………………………..5-6 Recommendations………………………………………………………………………………………………………………………6 II. Introduction………………………………………………………………………………………………………………………………..6 Purpose of the Evaluation…………………………………………………………………………………………………………..7 Design of the Evaluation……………………………………………………………………………………………………………..7 III. Methodology Methods of Data Collection………………………………………………………………………………………………………..7 On-Task Behavior Chart…………………………………………………………………………………………………………..7-8 Picture Survey………………………………………………………………………………………………………………………….8-9 Pre-and Post Modified KWL for Sixth Graders……………………………………………………………………………..9 Photo Interview Questions for Elementary Students…………………………………………………………………10 Interview Questions for Sixth Grade…………………………………………………………………………………………10 Limitations of Evaluation…………………………………………………………………………………………………………..11
  3. 3. Program Evaluation – Final Report IV. Results……………………………………………………………………………………………………………………………………..12 EvaluationFindings……………………………………………………………………………………………………………………12 Pre and Post -Sixth Grade Modified KWL Intrinsic and Extrinsic Findings………………………………….12 SEN K-2 Visual Intrinsic and Extrinsic Survey Findings………………………………………………………………13 On-Task Behavior Findings………………………………………………………………………………………………………..14 Photo Conference Questions Findings……………………………………………………………………………………..15 V. Conclusions and Recommendations…………………………………………………………………………………………15 VI. References………………………………………………………………………………………………………………………………..16
  4. 4. Program Evaluation – Final Report I. Executive Summary Purpose and Scope Interactive white boards (IWBs) are used in almost every classroom in this school district. They are used across content areas and have a variety of applications and uses. IWBs are currently used to promote learning in K-6 reading programs. The scope and purpose of this evaluation was to determine whether IWBs impact students’ attending behaviors and organizational strategies in K-6 reading programs. Student behavior logs, student interviews, SEN surveys and Photo “wh” questions were used to determine effectiveness of the IWBs. Based on the percentage data results of this evaluation, we found that the use of interactive white boards in K-6 reading programs does improve students attending behaviors and organizational strategies; therefore, this program should be maintained. Audience This evaluation took place from February 22, 2013 to April 5, 2013 in the KW School System and seeks to inform the stakeholders at KW School System on the impact of IWBs on K-6 reading programs. Data Collection and Analysis Data were collected through behavior logs, interview sessions, self-reporting and surveys of 58 students participating in K-6 reading programs. Behavior logs were tracked during white board segments, while daily observational data was collected by the evaluator. All of the evaluation participants were observed for attentiveness during their academic period. In the study, attentiveness was identified and operationally defined as active learning (i.e., looking at the speaker, focusing on the whiteboard, utilizing books, clipboards/dry erase boards, manipulating props and materials). Student’s off task behavior was identified and operationally defined as disengaged (i.e., playing with clothes, shoes, distracted, looking up at the ceiling, etc.). Attention to task was discussed at the beginning of each lesson and defined by the students as well. Data was recorded on a 5 minute-interval tally chart during a thirty-minute lesson. Active learners received a plus sign (+) and disengaged students received a minus sign (-). Weekly conferences were held for general education students, during which students completed photo interview questions assessing information retrieval. Participating sixth grade students completed pre and post evaluations to determine if there were correlations between IWBs and students’ utilization of organizational strategies. Participating AU students took part in a three session visual survey to determine if there were correlations between IWBs and students’ utilization of organizational strategies. Data herein is compiled into several charts, displaying results. These research methods were used with both an AU self-contained group of students and an inclusion group.
  5. 5. Program Evaluation – Final Report Findings The findings of the evaluation are listed below, categorized into four areas. Student Behavior Logs *(IWB = Interactive white board, WWB=with white board & WOWB=without white board) Daily 30 minute behavior logs were utilized by observers, at 5 minute intervals for six weeks to determine students’ on-task behavior. Both Group A and B were assigned numbers for student identification. Data on all eight students in Group A were recorded daily. Data on the 48 students in Group B was randomly taken due to class size. The qualitative data was tabulated weekly to measure on task behavior with an IWB and without IWB with a baseline of zero and a goal of 85%. Time on task was reported higher with the IWB in both the AU self-contained group (WWB=72%, WOWB=65%) and the inclusion group (WWB=78%, WOWB 52%). Although the researchers did not make their intended goal of 85%, there was active growth in each group. These data fluctuations were attributed to outliers of age and development of students, class-size, pre CRCT and post CRCT variables, and prior knowledge fluency. Interviews A modified version of the KWL format was employed before and after the study with the sixth grade inclusion group to measure the impact of IWB lessons on students’ confidence and expression of detail and interest. The qualitative interview questions were intrinsic and extrinsic in nature, and open-ended. The responses were rated on a 3-point rubric scale (1 – little detail, 2- some details, 3 – very detailed). Pre-Evaluation results were; averaged intrinsic responses, (1.75) and averaged extrinsic responses, (2.25). Post-Evaluation results were; averaged intrinsic responses, (2.5) and averaged extrinsic responses, (2.66). The results were compared and a percentage growth rate was tabulated. Intrinsic responses increased by .75 and extrinsic responses increased by .16. These data validated that IWB lessons contributed to students’ increased confidence and expression of detail and interest. SEN Survey A qualitative SEN survey was used with the AU self-contained group. The survey was visually designed (the employment of visual answers in place of verbal or written responses), to measure how the IWB influenced their intrinsic and extrinsic motivation in correlation with their implementation of organizational and retrieval strategies. An emotion survey rating scale (1-4) was employed for the intrinsic questions (1-2) and numbers were assigned for the students’ responses for the extrinsic based questions (3- 6). These data detected the students’ underdeveloped expressive and receptive language skills, which is common in their pervasive disorder, in their extrinsic based responses. Not unexpectedly, the percentages for extrinsic based responses yielded < -
  6. 6. Program Evaluation – Final Report .5 regression. In their intrinsic based responses, a significant negative correlation was discovered in the 1.75 percent growth rate. These data percentages displayed the participants dislike for conventional means of work tasks and an enjoyment for work tasks on or including the IWB. The data validates that the employment of the IWB impacts students’ intrinsic motivation, which theoretically, enables them to implement organizational and retrieval strategies. Photo “wh” Questions The study included utilizing photographs to assess students’ engagement and retrieval process during IWB lessons. Once a week students were shown a picture of their class during an IWB reading large group lesson. All students were asked a combination of questions, some open-ended and story element questions. They were instruction to give detailed responses. The responses were rated on a 3-point rubric scale (1 – little detail without explanation, 2- some details without explanation, 3 – very detailed with descriptive explanation). The results were averaged each week and those averages were divided into six. The final results were Group A (2.25) and Group B (2.5). Interestingly, both groups remained relatively consistent with neither displaying a significant increase or significant regression. With Group A, the photo questions showed that, although the students could recognize themselves in the pictures and familiar stories, they were unable to express detailed accounts. With Group B, the students exhibited engagement, maturity and enjoyed detailing their photos. Recommendations 1. IWB lessons should be implemented in K-6 reading programs, if they are not already. 2. K-6 reading programs already incorporating IWB lessons should do so at an increased level. 3. KW School System should encourage increased use of IWBs in reading programs. 4. Research should be continued to determine the effectiveness of IWBs on other academic programs. II. Introduction The majority of classrooms in KW School District contain Interactive White Boards (IWBs). These white boards are often used in reading programs. This program evaluation sought to determine the effectiveness of IWBs in K-6 reading programs. Two evaluation questions drove this study: a. Can the incorporation of an interactive white board impact students attending behaviors during guided reading? b. Can the incorporation of an IWB increase students’ (general education and SWD) utilization of organizational strategies?
  7. 7. Program Evaluation – Final Report Purpose of the Evaluation Stakeholders agreed that the impact of IWBs should be evaluated in K-6 reading programs. These findings will be shared with faculty and staff to determine future use of IWBs in K-6 reading programs, as well as the use of IWBs in other content areas. Design of the Evaluation Two student groups were included in this study. Group A consists of eight self-contained autistic students. Group B consists of 48 students in inclusion classrooms. Of the 48 students in group B, 19 receive gifted services, 10 receive special education services and one receives both gifted and special education services. Data collection methods were the same for both groups. Behavior logs were tracked during white board segments, a modified version of traditional KWL (what you know, what you want to know and what you have learned) was utilized for interview questions (Group A was given super symbols to answer interview questions), and photo questions were used with both groups. III. Methodology Methods of Data Collection Data were collected from February 22, 2013 to April 5, 2013. Behavior logs were tracked for the duration of the study, six interview sessions took place from March 1st to April 5th , self-reporting for sixth grade students took place on February 22nd and April 5th , and SEN surveys were distributed three times, March 1st , March 15th and March 29th . Figure A. On-Task Behavior Chart The chart below was used to record on-task behavior with and without an IWB on all participants. With IWB Days of the Week Monday Tuesday Wednesday Thursday Friday Totals Average Number __ +/- Number __ +/-Number __ +/- Number __ +/- Number __ +/-
  8. 8. Program Evaluation – Final Report Without IWB Days of the Week Monday Tuesday Wednesday Thursday Friday Figure B- Picture Survey The picture survey below was used for elementary students. Picture Survey Questions: Student Number____ Time______ Date________ When I use scissors to cut out story pictures then glue stick to glue them in a square, I feel……. 1 2 3 4 When I use the white board and pen to move story pictures to a square, I feel…….. 1 2 3 4 I can find the title of a book and the author using a………… 1 2 3 I can see the bigger words, story characters, story settings, when I look at… 1 2 3 Totals Weekly Average Number __ +/- Number __ +/-Number __ +/- Number __ +/- Number __ +/-
  9. 9. Program Evaluation – Final Report When I watch stories on the white board and see the big words and pictures, I…… 1 2 3 It is easier to use my story checklist at my seat looking at the story on the whiteboard or book 1 2 Figure C - Pre-and Post- Modified KWL for Sixth Graders Prediction and What I Know- Bubble Head (Intrinsic) Prediction and What I Know-verbally respond (extrinsic) * Will you learn anything using the interactive whiteboard (IWB)? 1 – little detail 2- some details 3 – very detailed • What new reading skill will learn a using the IWB? 1 – little detail 2- some details 3 – very detailed • Will you be patient taking turns at the board? 1 – little detail 2- some details 3 – very detailed • Do you think the IWB will change the way you think about reading? How? 1 – little detail 2- some details 3 – very detailed • Have you ever put your own work on the IWB? 1 – little detail 2- some details 3 – very detailed •Who would you tell in another school: a teacher/child to use the IWB? 1 – little detail 2- some details 3 – very detailed • Do you think other children/teachers/parents learn with the IWB? 1 – little detail 2- some details 3 – very detailed • Do you think using the IWB for reading will be a good idea? Why? 1 – little detail 2- some details 3 – very detailed • Do you think it won’t be fun to use the IWB? 1 – little detail 2- some details 3 – very detailed • Would you feel embarrassed to show anyone your work on the IWB? Why? 1 – little detail 2- some details 3 – very detailed • How do you feel about working on the IWB? Why? 1 – little detail 2- some details 3 – very detailed • What grade would learn the best using the IWB?1 – little detail 2- some details 3 – very detailed *How did answering these questions make you feel? 1 – little detail 2- some details 3 – very detailed
  10. 10. Program Evaluation – Final Report Figure D- Photo Interview Questions for Elementary Students Interview questions Elementary Student Number:____ Time:_____ Date:_________ Candid pictures of the students during their reading segment were employed to ask the following questions…. What are you thinking in this picture? 1 – little detail (Student did not remember specific details) 2- some details (Student remembers 1-4 specific details) 3 – very detailed (Student remembers and describes specific details) Do you remember the name of the story? 1 – no title 2- half title 3 – full title What was your favorite part of that story? Why? 1 – little detail without explanation 2- some details without explanation 3 – very detailed with descriptive explanation Total_____ Average______ Figure E - Interview Questions- 6th grade Interview questions-6th Student Number:____ Time:_____ Date:_________ What are you thinking about in this picture? 1 – little detail 2- some details 3 – very detailed When you look at the picture: Do you remember the day off the week? 1 – little detail 2- some details 3 – very detailed What lesson is it? 1 – little detail 2- some details 3 – very detailed What is going on in the story? 1 – little detail 2- some details 3 – very detailed Who was the main character? 1 – little detail 2- some details 3 – very detailed Total_____ Average______
  11. 11. Program Evaluation – Final Report Limitations of the Evaluation Although IWBs are used in various programs throughout the school, this study focuses on their effectiveness in reading programs only. The number of student participants in this study is also limited. Due to time constraints, a larger number of students could not be assessed. Data analysis tables were not preformed due to the limited sample population and relevant past research was given honorary mention in the appendix for inspiring and guiding the study, but not directly cited in the evaluation. IV. Results Findings Research showed that IWBs positively impact students attending behaviors and organizational strategies, as displayed in Pre-and Post-Sixth Grade Modified KWL Chart, SEN K-2 Intrinsic and Extrinsic Survey chart, on-task behavior chart and photo questions survey chart, found in the appendices. Figure F - Pre and Post -Sixth Grade Modified KWL Intrinsic and Extrinsic Findings 0 0.5 1 1.5 2 2.5 3 1 2 3,2,1AveragedRubricScores 1 2 Pre 1.75 2.5 Post 2.25 2.66 Pre and Post Sixth Grade KWL Results First and Last Week
  12. 12. Program Evaluation – Final Report The Pre and Post KWL Interview data were utilized to determine IWBs’ impact on students’ personal and academic (intrinsic and extrinsic) organizational strategies. These data illustrated a positive correlation between students’ intrinsic and extrinsic organizational strategies with the use of IWBs in reading programs. Figure G - SEN K-2 Visual Intrinsic and Extrinsic Survey Findings Elementary students’ intrinsic scores significantly improved with the utilization of IWBs in their reading programs. These data can signify that the students are personally invested in their academic tasks. 0 0.5 1 1.5 2 2.5 3 3.5 1 2 3 3,2,1AveragedRubricScores 1 2 3 Group A SEN Intrinsic 3.25 3 3.5 Group A SEN Extrinsic 1.5 2 1.5 SEN K-2 Visual intrinsic and Extrinsic Survey Number of Survey Weeks in
  13. 13. Program Evaluation – Final Report Figure H - On-Task Behavior Findings All groups showed an increase in on-task behavior with IWB lessons. 0 0.5 1 1.5 2 2.5 3 1 2 3 4 5 6 1 2 3 4 5 6 Group A 2 2 2 2 2.5 3 Group B 2.25 2.5 2.25 2.75 2.75 2.5 Photo Conference Questions Group A Group B Week Number 3,2,1AveragedRubricScores
  14. 14. Program Evaluation – Final Report Figure I - Photo Conference Questions Findings V. Conclusions and Recommendations Because research showed improved on-task behavior and improved organizational strategies with the incorporation of IWBs into K-6 reading programs, the following recommendations are made: 1. IWB lessons should be implemented in K-6 reading programs, if they are not already. 2. K-6 reading programs already incorporating IWB lessons should do so at an increased level. 3. KW School System should encourage increased use of IWBs in reading programs. 4. Research should be continued to determine the effectiveness of IWBs on other academic programs. 5. Research should further explore the correlation between IWBs utilization in the classroom and its impact on students’ beliefs about learning and metacognition. Successful implementation of this study, in conjunction with past research and other school improvement efforts, promises positive correlation with IWB’s and the students’ initiation and continued self-regulation of attending (active learning) during lesson time, which research states, improves students’ academic achievement and self-efficacy. 0 0.5 1 1 2 3 4 5 6 1 2 3 4 5 6 Group A NWB 0.56 0.63 0.59 0.65 0.71 0.74 Group A WB 0.62 0.67 0.73 0.77 0.75 0.81 Time on Task: Group B without IWB 0.4 0.33 0.55 0.65 0.7 0.5 Time on Task: Group B with IWB 0.75 0.7 0.7 0.85 0.9 0.7 Time On Task - with and without IWB Week Number AveragedPercentages-TaskOnTask
  15. 15. Program Evaluation – Final Report Reference Page Honorary Mention The following are articles that supported and gave perspective to the Evaluation Study. Carnahan, C., Williamson, P., Hollingshead, A., & Israel, M. (2012). Using technology to support balanced literacy for students with significant disabilities. Teaching Exceptional Children, 45(1), 20-29 Frauenberger, C., Good, J., & Keay-Bright, W. (2011). Designing technology for children with special needs: bridging perspectives through participatory design. CoDesign, 7(1), 1-28. doi: 10.1080/15710882.2011.587013 McKenna, M., Labbo, L. & Reinking, D. (2003) Effective use of technology in literacy Instruction, in: L. Morrow, L. Gambrell & M. Pressley (eds), Best Practices in Literacy Education, 2nd edn. (New York, The Guilford Press). Solvie, P. (2007). Leaping out of our skins: postmodern considerations in use of an electronic whiteboard to foster critical engagement in early literacy lessons. Educational Philosophy and Theory, Vol. 39, No. 7, 2007. doi: 10.1111/j.1469-5812.2007.00312.x

×