Designing an evaluation of a tertiary preparatory program sounds
Upcoming SlideShare
Loading in...5

Designing an evaluation of a tertiary preparatory program sounds







Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Designing an evaluation of a tertiary preparatory program sounds Designing an evaluation of a tertiary preparatory program sounds Presentation Transcript

  • Designing an Evaluation of a Tertiary Preparatory Program within the University Context By Ravinesh C Deo
  • Presentation Overview
    • Introduction to a Program Evaluation?
    • Benefits of an Evaluation
    • Problem Statement
      • Challenges during First Year of Tertiary Study
      • Preparatory Program in Schools
    • Designing an Evaluation
    • Ethical Factors to Consider
    • Reporting Back Mechanisms
    • Conclusion
      • Limitations of Current Plan
  • What is a Program Evaluation
    • A process that examines success of an educational event, syllabus design, content, implementation or achievement of objectives.
      • assesses merits of a product, a program or a practice
    • Is conducted using a multi-faceted and systematic approach
      • focus on design, implementation and effectiveness
    • Outcomes enable educational practitioners to gather evidence whether their actions and decisions are appropriate
    • Helps determine whether their program fulfils aim and ethos of their academic institutions
  • The Tertiary Preparatory Program
    • The Open Access College of University of Southern Queensland designed a teaching and a learning program known as the “Preparatory Program in Schools”.
      • designed for Year 11 students to complete a set of tertiary courses during their final school years
      • fulfils commitment of the College to supporting students who are not able to make informed decisions about career path
      • creates avenues for various career options
  • Why Evaluate the Tertiary Preparatory Program in Schools?
    • A new program so a need to assess the educational quality
    • An opportunity to identify components that require improvement
    • Outcomes may be relevant to the success of the program in areas like
      • policy development
      • decision-making
      • funding opportunities
      • partnerships between the institution and community.
    • Promote confidence that academic programs are being monitored, reviewed, modified and researched
    • Identify challenges faced by first year students in adapting to university teaching and learning environments
  • Challenges Faced by First Year Students
    • School leavers face challenges during first year studies, as evidenced by
      • Nationally attrition rate of 28.6% in 2002
      • Queensland attrition rate of 31.1%
      • After first year, a drop to 10 % (national) & 11.5 % in QLD
      • (DETA, 2004).
    • Peel (1999) identified some challenges as
      • inadequate preparation for independent style of learning
      • lack of prior exposure to tertiary environments
      • gaps between course structures and expectations of university and school
      • “ expectation” that university will deliver life challenges
      • diversity of teacher-student relationship and a more formal relationship with lecturers
  • Performance Linked to Lack of Motivation?
    • Research Evidence?
      • Lowe and Cook (2003) showed 20% of students could not adjust to academic and social demands of a university.
      • Hillman (2005) found that students from low socio-economic backgrounds have difficulties in balancing work & study commitments.
    • These could be worsened by poor motivation and lead to a diminishing of interest in studying
    • As a result –performance during first year at university becomes poor.
  • The Government’s Decision
    • The Bradley Review examined our higher education sector.
      • incorporated recommendations on encouraging participation of persons from low socio economic backgrounds in higher education.
      • focussed on enhancing partnerships between schools, institutions and communities through outreach activities.
    • Declared that by 2020, 20% of tertiary enrolments are expected to be from low socio-economic regions
    • (Bradley, Noonan, Nugent, & Scales, 2008).
  • Response of Open Access College
    • Committed to social inclusion for supporting students who are unable to make informed decisions about prospective careers.
    • Provided pathways to those who miss on education opportunities
      • Preparatory Program in Schools
      • Created alternative pathway for students to complete two tertiary level courses during final school years.
    • Designed to increase participation from Ipswich & Moreton region
      • identified as Socio Economic Disadvantaged
      • Only 35.7% and 19.4% respectively, of Year 12 students enter tertiary institutions (DETA, 2008; Australian Bureau of Statistics, 2006)
  • Designing an Evaluation
    • Program evaluations may be
      • objective-oriented
      • expertise-oriented
      • management-oriented
      • naturalistic/participant-oriented
      • (McNeil, 1996).
    • To serve a number of purposes, we propose a mixed model whose choice is based around four main objectives
      • (a) assess attainment of educational standards by students,
      • (b) investigate how effectively students integrate into a tertiary learning environment,
      • (c) determine whether students experience increases in motivation for study
      • (d) assist students in setting up a prospective career path
  • Acquisition of Data
    • Question 1 : What are the effects of the program on motivation for studies, development of communication, mathematical skills and decision-making on prospective career paths?
    • Purpose of the Question: Assess
    • changes in student’s learning profiles (mathematical reasoning & communication),
    • increases in motivation for studies and commitment towards action-planning and scheduling study activities,
    • integration into a university teaching and learning environment .
    • To Address Question 1: Data will be collected using 4 main avenues
    • Academic Performance
      • The Preparatory Program in Schools has courses with 4 assignments each.
      • Based on attainments in each, marks will determine how well students are adjusting to university learning environment.
      • Continuous monitoring will ensure specific trends in learning attributes are recorded progressively.
      • Assignments have a “learning diary” and “summative essay”
        • examine plans for future careers.
    • Weekly Journal Entry
      • As part of non-graded assessment, students will submit weekly journals
      • Will provide an opportunity to self-assess their learning journey and the freedom to provide their own perspectives.
      • Will provide information unique to individual students (i.e. personal learning attributes).
    • Covert Observations
      • used to collect data on student’s learning profiles within a natural learning environment
      • learning behaviours observed by the facilitator, allocated a rating on a numerical scale from 1-10.
      • frequency will be weekly
    • Focus Groups
      • pre-defined student-focus groups
      • stimulate discussion on a set of topics from each course.
      • examine study management/problem solving abilities
      • frequency twice a semester
    • Question 2 : What are the effects of the program on professional development of the College staff?
    • Purpose of Question:
      • Obtain staffs’ perceptions on the success of the program
      • Reveal challenges (e.g. resources) the institution is facing
      • Examine whether the program is having a positive impact on professional development
      • Create options for further opportunities for staff that might help improve the program delivery
    • To Address Question 2: Data will be collected using 2 main avenues
    • ff Surveys
      • describe general experiences
      • frequency will be twice annually.
    • SWOT Analysis ( Staff present their roles/responsibilities)
      • Strengths (Internal): identify new learning or pedagogical skills
      • Weakness (Internal): identify challenges (e.g. anxiety with “new” cohort)
      • Opportunities (External): opportunities available (e.g. ICT training,)
      • Threats (External): assess the constraints (e.g. workloads)
    • Question 3 : W hat extent did the program succeed in attracting community confidence?
    • Purpose of Question: recognise the community’s role & participation
    • Parent-Teacher Interviews
      • random batch of parents invited for interview on monthly basis.
      • examine any growing community support for the program
    • Influences Questionnaire
      • based on influences questionnaire (Taylor & Bedford, 2004)
      • questions centred on factors that influence a parent in withdrawing their child or continuing
        • 1 to 5 scale (1 = not interested, 2 = slightly, 3 = unsure, 4 = moderately, 5 = extremely interested).
  • Validity and Reliability
    • Multi-Model Approach:
      • several modes of collection & merging results can lead to a valid, reliable and diverse construction of realities (e.g. Patton, 1997).
    • Student Dropout:
      • reduction in numbers can bias our results
      • to overcome this, we will compare characteristics of those students who have dropped out with those who remain.
      • if the two groups are not significantly different in terms of characteristics, then dropouts did not affect our results
      • (Heckman, Smith, & Taber, 1998)
    • Naturalistic Methods:
      • One example is use of covert observations
    • (Lynch, 1996).
    • Well-defined & Systematic Procedure:
      • can be replicated in future, thus ensuring the reliability .
  • Ethical Factors to Consider
    • Should be considered to avoid harm (Sanderson & Mazerolle, 2008).
    • Ethical Clearance sought from University Ethics Committee.
    • Principle of Informed Consent (Evans & Jakupec, 1996).
      • all participants educated about the evaluation, purpose and potential benefits (Wiles, Crow, Heath, & Charles, 2006).
    • Confidentiality of Information :
      • Using pseudonyms to avoid name disclosure (Richards & Schwartz, 2002).
    • Avoiding Loss of Life-Time Opportunities :
      • for example falling behind contemporaries, being graded as unsuccessful or losing their career as an outcome of evaluation (Israel & Hay, 2006),
      • outcomes not used to process admissions in any other academic programs.
    • Non-disclosure of Findings – no information disclosed to any third party
    • Avoid Psychosocial Consequences :
      • e.g. loss of self-esteem, self-concept or confidence (Bibby, 1997).
      • done by ensuring challenged student’s cognitive limitations are not disclosed.
    • Privacy to Information:
      • granting freedom to deny response to any particular question (Sharp, 1994).
      • avoid discussing characteristics that discovers the subject’s identity
    • Cultural Sensitivity:
      • exercised to protect identity of community and cultural denomination (Hafford & Lyon, 2007).
    • Data Security:
      • deleting all subject identifiers
      • security devices such as passwords to protect electronic data .
  • Reporting Back Mechanisms
    • Progress Reports :
      • fortnightly reports on changes in students’ learning profiles
    • Newsletters : parents and school sent monthly newsletter
      • ensuring that no specific results are disclosed to students to avoid potential bias , “Hawthorne effect” (Kuper, Lingard, & Levinson, 2008).
    • Consultative Meetings : progress reports discussed in consultative meetings
    • General Assembly Meeting : merged in a master document
      • Tabled in a College Assembly to gather insights and strategic course of actions such as
        • restructuring the program
        • providing professional development for staff
        • promoting the program across wider community .
  • Limitations of Evaluation Plan
    • Conditional Conclusions : Outcomes true under specific conditions that are couched in probabilities rather than absolute certainty.
    • Time Frame : proposed 2 semester timeframe not be adequate to conduct a comprehensive assessment.
    • Predisposition : selection based on interview so all interested students may be eager to join.
      • may attract those who are predisposed to a positive outcome
      • measuring changes in profiles may overstate the achievements
    • Maturation : Does not take into account effects of maturation.
      • events outside the program cause changes in knowledge or behaviours
      • growth of abilities due to maturation not quantified easily
      • a positive outcome due to maturation treated as a plus of the program
    • Overcome These?
      • Increase the length of study, 2-5 year period
      • Use counter-control mechanisms based on previous year’s findings
  • References
    • Australian Bureau of Statistics. (2006). Socioeconomic indexes for areas. Retrieved April 20, 2011, from
    • Bibby, M. (1997). Introduction: Education research and morality. In M. Bibby (Ed.), Ethics and education research (Review of Australian research in education no. 4) . Melbourne, Vic: Australian Association for Research in Education.
    • DETA. (2008). Next step 2008: A report on the destinations of year 12 completers from 2007 in Queensland . Queensland: Department of Education, Training and the Arts.
    • Evans, T., & Jakupec, V. (1996 ). Research ethics in open and distance education: Context, principles and issues. Distance Education, 17 (1), 72-94.
    • Hafford, C., & Lyon, K. (2007). The Challenges of Conducting Evaluation in Tribal Communities . Washington, D.C: Office of Family Assistance.
    • Heckman, J., Smith, J., & Taber, C. (1998). Accounting for dropouts in evaluations of social experiments. Review of Economics and Statistics, 80 (1), 1-14.
    • Hillman, K. (2005). The first year experience: The transition from secondary school to university and TAFE in Australia . Victoria, Australia: Australian Council for Educational Research.
    • Israel, M., & Hay, L. (2006). Research ethics for social scientists: between ethical conduct and regulatory compliance . London: Sage.
    • Kuper, A., Lingard, L., & Levinson, W. (2008). Qualitative Research Critically appraising qualitative research. British Medical Journal, 337 , a1035
    • Law, H., & Cook, A. (2003). Mind the gap: are students prepared for higher education?. Journal of Further and Higher Education, 27 (1), 53-76.
    • Lynch, B. (1996). Language program evaluation . Cambridge: Cambridge University Press.
    • McNeil, J. D. (1996). Curriculum: A Comprehensive Introduction (5th ed.). Los Angeles: Harper Collins College Publishers
    • Patton, M. Q. (1997). Utilization-focused evaluation: the new century text (3rd ed.). Thousand Oaks, California: Sage.
    • Peel, M. (1999). Where to now? Transition from Secondary to Tertiary: A Performance Study Higher Education Series (Vol. 36, pp. 13-16).
    • Retrieved April 20, 2011, from
    • Richards, H. M., & Schwartz, L. J. (2002). Ethics of qualitative research: are there special issues for health services research? . Family Practice, 19 (2), 35-39.
    • Sanderson, J., & Mazerolle, P. (2008). An evaluation of the book. Retrieved April 27, 2011, from
    • Sharp, C. A. (1994). What is Appropriate Evaluation? Ethics and standards. Evaluation News & Comment, 3 (2), 34-41.
    • Wiles, R., Crow, G., Heath, S., & Charles, V. (2006). Anonymity and Confidentiality ESRC National Centre for Research Methods, NCRM Working Paper Series (Vol. 2/06.).