Final project program evaluation module 8Document Transcript
Purpose and Focus of EvaluationThe purpose and focus of the evaluation is to better understand the results of studentlearning in a multiple discipline areas with the use of technology. The evaluation will attempt toevaluate how project based learning improves student learning of technology tools andmaterials. This evaluation is a formative evaluation which aims to prepare needs assessment aswell. Since the program is new it will need to make necessary adjustments as the programgrows. This program has been developed to allow students to make independent decisions onhow they can learn and express themselves creatively with technology.Description of ProgramThe program to be evaluated is a high school technology requirement. This program is arequired course in which students complete the course using their choice of subject area (science,English, social studies) develop a project using technology. This course is proposed to start in the2011 school year. The course is taught by technology department teachers and utilizes teachercollaboration from multiple subject areas. Students need to develop their own project conceptsand what technology they will use to complete the project.What stakeholders will be informed?The stakeholders who should be informed are the Director of Technology, Principal of theHigh school and prominent and effected department heads. These stakeholders should beinformed of the results of this evaluation in order to understand whether the program they areusing promotes certain outcomes.
Evaluation of Capabilities and Risks of EvaluationThe capabilities and risks of doing this evaluation are relatively small. The evaluation willrequire as few as three individuals to work as individuals to administer interviews, collect videodata and administer testing procedures. The major investment in the completion of thisevaluation is time for data collection and organization. The cost of having individualadministering the testing and surveying materials is anticipated to be minimal.What are the political ramifications?The political ramifications for this evaluation are that it could empower the technologydepartment to have a larger role in curriculum development or it could hurt the need fortechnology in the school curriculum. The results of the questions could provide solid data thattechnology learning improves project based learning techniques. This program could also showthe weaknesses of student’s technology learning.Description of Approaches and Models Used“The distinguishing feature of an objectives orientated evaluation approach is that thepurposes of some activity are specified, and then the evaluation focuses on the extent in whichthese purposes are achieved.”(Fitzpatrick,Worthen,Sanders,2004) It is the object-orientatedapproach in which my evaluation will seek to determine the extent of student learn acrosscontent areas, skill levels and technology projects. The key questions of the evaluation will seekto define specific goals and learned skills.
Object Orientated Approaches to Key Questions1. How does the WHS tech requirement increase the student use of technology across the“content areas”?Objectives-orientated approachThis question requires that specific data regarding students using different content areas inorder to complete their technology projects. This question gets data pertaining to the specificcontent areas students will use. This question will require on qualitative and quantitative datacollection. The described content areas like English, math or Social Studies will be asked in bothquantitative questions and in qualitative interviewing scenarios.2. Based on the standards of the course requirement will students engage more in the projectbased learning?Objectives –orientated approachThis question will measure student’s level of engagement and learning with project-orientatedschool work by having to use technology. This question will be answered primarily in qualitativeinterview questioning. This specific question is aimed primarily at teachers but there is ananticipation that all participants will be asked to reflect on their experiences with subject matterssimilar to project based learning. There will be quantitative data collected from likert surveys tomeasure the tendencies of learning styles. One of the anticipated learning styles anticipated in thequestions will be “Project Based”.3. Will the use of project based learning increase student use of technology?
Objectives-orientated approachThis question will measure the student’s use of technology before and after the beginning ofthis program. To answer this question most definitively will be to create test score summariesfrom the pre and post technology skills test. This data can be constructed to show the growth oflearning as well as possible areas of limited learning.With Qualitative data students can express in interviews or focus groups their perceptions oflearning. In this forum students will hopefully respond to their understanding of their learningand expectations. Students and teachers can reflect on specific uses of technology and scenariosin which they felt as though they expanded their knowledge base.Who is the Audience or Stakeholders?The audience for this evaluation would be the High School Principal, Director of Technology,department heads of different departments, Assistant Superintendant of Curriculum. The coregroup of teachers and administration and committee to help develop and implement the programwould also hold interest. The data from the evaluation will help the stakeholders make changesin the curriculum or implementation of the program.The stake holders will also be part of the formative evaluation of the program as their inputinto future evaluations and goals will be taken into account. It will be important to worksuccessfully with the stakeholders in order to ensure their participation and interest in continuallyevaluating the program.
An important factor in this evaluation is the school community of parents, administrators andteachers. In Winchester, Massachusetts the affluent community cares deeply about the educationof the students and the programs they are involved with. “Dialogue with multiple audiences alsoclarifies the reasons behind an evaluation.”(Fitzpatrick,Worthen,Sanders,2004) involving theseaudiences will empower the evaluation and show clear reasons for the evaluation on all sides.What are the key questions?The evaluation questions are based on the outlined focuses of the program. Underagreement between major stakeholders (High School Principal, Technology Director, HighSchool Department Heads) the focused areas of the High School Technology Program will teachtechnology courses which allow students to take advantage of multiple subject disciplines,creating technology final projects and learning specific technology skills. Having the keyquestions aligned with the stakeholders thinking should enhance buy in from theadministration.The Big Questions1. How does the WHS tech requirement increase the student use of technology across the“content areas”?2. Based on the standards of the course requirement will students engage more in the projectbased learning?3. Will the use of project based learning increase student use of technology?
Summarized Researched Findings from a Project Based Learning ArticleThe research article attempts to make sense of current data and definitions of project basedlearning defined in the article as “PBL”. The article clarifies what project based learning ispredominantly defined as and also explains that part of the problem with assessing project basedlearning is that there have been different definitions of what it entails and how it has beeninstructed in various contexts.One of the key concepts of instruction with “PBL” is the “project as the central teachingstrategy.”(Thomas,2000) This means that all work and instruction is aimed at completing theproject not teaching subject areas in a central way only as needed. The projects are studentdriven and when they are completed are realistic and not school-like. (Thomas,2000)Conclusively PBL has students constructively investigate their question and use problem solvingto follow the investigation to completion.The article describes common characteristics of project based learning as a “model whichorganizes learning around projects” (Thomas,2000). This model “culminates in realistic projectsor presentations which use an authentic driving question and the use of tools of technology.”(Thomas,2000) These open descriptions and programs which have adjusted the variables of PBLhave made it difficult to define PBL to one type of instructional models.The “Expeditionary Models” which are similar to the predominant one described in the articleas Project Based Learning (PBL) has been shown to increase learning in content areas as well asskill based learning. One example is the researches conducted at the New American SchoolsDevelopment Corp. Using “Expeditionary Learning” (PBL) students have improved standardizedtests scores on the Iowa Basic Skills in Dubuque, Iowa from “Well Below Average” to “District
Average”. The magnitude of change in schools through the schools in Dubuque, Iowa was from15%-90%. (Thomas,2000)This research substantiates the evaluation of the learning of technology skills in the RequiredTechnology Course Program with the use of Project Based learning Initiatives. Students in theprogram will be self directed learners who will problem solving skills in order to completeprojects. The evaluation of the learned technology skills will determine if the PBL approach isappropriate and successful in the Required Technology Course context.Key Question AnalysisThe key questions will be addressed with a series of data collection techniques which willcollect quantitative and qualitative data. Each type of data is necessary to best assess theprogram’s success at teaching the skills that need to be learned with technology but all theexperience and process of the students projects in the program.The qualitative questions will be addressed primarily in interviews and focus groups. It isimportant to collect this data because it will give insight to the learning process involved withspecific student’s projects as well as their happiness with the program.Quantitative questions will be asked in the form of likert surveys and test summaries. Thesequestioning methods are necessary to understand in clear numbers the learned skills andknowledge as well as percentage data which can reflect the student happiness and perceivedsuccess with the operation of the program.Research Methods & Feasibility Issues
How does the WHS tech requirement increase the student use of technology across the“content areas”?Research Method: Students will take a likert scale survey which will assess their inclusion ofother subject disciplines with technology learning before and after their participation in thecourse. These questions will ask students to choose a content area in which they worked and tryto develop an understanding of their selection for specific content areas. The variable with thisquestion is if the students who are using technology across disciplines are learning new skills orare they just applying it differently to a new subject area.Based on the standards of the course requirement will students engage more in the projectbased learning?Research Method: Students will be given a likert survey which assesses the student’s likelihoodto complete a long range project using technology before and after their participation in theproject. Students will also be asked to define specifically their work ethic in projects. Teacherswill be asked to measure student reaction to multiple step project assignments. Administratorsmay be asked to express their opinion on the showcasing of multi-step projects. A variable in thisquestioning is individual’s completion of projects. Some may complete the project successfullybut not learn new tools or do it in a project based way. These variables allow for new questionsabout the learning methods.Will the use of project based learning increase student use of technology?
Research Method: Students will be surveyed on their use of technology habits before and aftertheir participation in the course. This question is looking for a growth of technology tools beingused as well as the student’s knowledge of technology tools. The pre and post test of technologyskills will give in a clear presentation of quantitative data the stronger and weaker areas of thestudents technology knowledge gained over the course of the program. A variable for thisquestion is the normal difference in some students learning compared to others and how thismight affect the data.Data TypesThe quantitative data that I will receive will be in the form of questionnaires and likertsurveys from primary sources. These questionnaires should reflect in quantitative percentages thehabits, confidence, skills of students use of technology, project based learning and independentwork skills and experience.There is qualitative data to be collected from interviews and focus groups. This data will alsobe a primary source. This data will be from students, teachers, administration and community.This data will allow individuals to reflect openly on their experiences with the program. Thequestioning will be open ended and encourage personal expression with inclusions of scenariosand portrayals of the sources experience.Questionnaires-Open question questionnaires, for those who cannot be interviewed. Data willprovide open answers to questions regarding participation and final outcomes in the program.Test Data-Test scores and summaries which will provide percentage data of learned skills intechnology.
Interview transcriptions-Transcribed data of interviews of participants providing experiences andscenarios.Interview videos-Interview data which can be used for continual viewing and for finalpresentation.Likert Surveys-Surveys to determine opinions and feeling pertaining to student involvement withthe program as well as their learning tendencies.Measurement of Quantitative & Qualitative DataThis evaluation will be a formative evaluation which will provide data of the success of majorcomponents of this plan. This evaluation will provide quantitative data which will show thestudents tendencies with technology and learning on their own. It will be important that studentsare able to provide a broad range of responses in regards to their use of technology. It will alsobe important that a broad range of students are surveyed in order to gain quality data whichincludes students who willingly take technology courses and those who may have a limitedexperience with technology learning.Feasibility Issues of Evaluation: Data Sources, Sampling Strategy & Validity IssuesThe data sources for this evaluation will be prospective students for the high schooltechnology required course. They will be randomly sampled. Some of the validity issues maydevelop with students who have taken no technology courses or those students who have takenother high school courses or have been using technology regularly in their classes to completeprojects.
The sampling strategy for the data is to use 100 participating students as well as allparticipating administration and teachers. The sampling size allows for focus groups as well asindividual interviews. The sampling size also allows for complete testing to be completed in arelatively short time.Potential validity issues exist with students being able to discuss their experiences before testsand interviews. Potential bias and validity issues exist with the evaluator working within theprogram being evaluated. Considering the relatively short testing times the data collection shouldshow little to no validity issues. The evaluator will hire individuals to administer tests and directfocus groups and interviews.The Content, Audience and Focus of the Final ReportThe presentation of this report would be a presentation night specifically at the high schoolin which everyone could receive a copy of the report and watch a multimedia presentationwhich outlined some of the major parts of the report.The multimedia presentation would use a series of videos of student experiences and storiesabout their participation in the program. These “experience stories would give real life scenariosand portrayals of their learning in technology and the skills they gained. The video can also showsome of the qualitative interview data given from major stakeholders and express some of thekey goals of evaluating the program.A key piece of the presentation will be to show some of the pertinent quantitative data in aPowerPoint presentation. This will reveal the results of the evaluation and then open up thepresentation for a question and answer period. An open and honest presentation of the experience
of all of the students involved will be very powerful with the audience. The expected audiencewould be students, parents, teachers, administrators and community members.Time and Task Management Plan VisualTime and Task Plan Visual: High SchoolTechnology RequirementSeptember 1, 2011Pre-Evaluation Site Visit: Evaluate site contextof learning, resources, anticipated learningenvironment.Participants: Students, Teachers,Administration, SchoolTime Period: One MonthPrior to evaluation start.September 15,2011Pretest Of Technology Skills: Technology skilltest based on National Technology Standards.Participants: Participating StudentsTime Period: 2-3 DaysSeptember 19,2011Likert Surveys Pre-test: Evaluate student,teacher, administration opinions, attitudesand dispositions with technology curriculumlearning and process.Participants: Teachers, Administrators,Time Period: 2-3 Days
StudentsSeptember 23-December 10, 2011Continual Site Visits During ProgramTime Period: Once aWeek for durationOf program.December 12-13,2011Interviews with Teachers, AdministratorsTime Period:2 DaysDecember 13-15,2001Interviews with StudentsTime Period: 3 DaysDecember 16-18, 2011Focus Groups Meet 3 SectionsSubjects: Project based learningTechnology skillsContent AreasTime Period: 3 DaysJanuary 2-20Data Analysis and Synthesis Time Period: 3 wks.February 20th: Final Presentation of Findings Time Period: 2 Days
ReferencesFitzpatrick, S.,Sanders,J.,Worthen,B. (2004). Program Evaluation:Alternative Approaches and PracticalGuidelines 3rd Edition (. Boston: Pearson Education.Thomas,J. W. (2000). A Review Of Research On Project Based Learning. San Fafael,California: TheAutodesk Foundation.