1. NIJESH SOLK MWIL
START GYAMYATH
(Save us from the time of trial)
From Austerity to Prosperity and back again
ESAI CONFERENCE 2013
M.Brown, G.McNamara, J.O’Hara
Centre for Educational Evaluation (CEE), DCU
2. FOCUS OF THE STUDY
This study examined the changing landscape of evaluation
policy and practice in the ROI and NI since the inception of
the Stanley letter 1831 where evaluation in the form of
school inspection and self evaluation aims to promote
and by turn of phrase improve the quality of education
provided in schools.
From this, a framework for the mutual terms of
co-existence between IE and EE is proposed.
Theoretical Assumption
‘Both systems should exist because we need both and because they might
even benefit from each other’
(Nevo p.6, 2002)
Centre for Educational Evaluation (CEE), DCU
3. OUTLINE OF THE STUDY: MAY 2009 - MARCH 2013
Phase 1 :- Tentative Schemata
Review of the literature on the rise of evaluation in education and
the different international models of evaluation that exist, coupled
with an analysis of principal and inspector interviews in the 19th
and 20th century.
Phase 2: - Quantitative Analysis
All Island survey of school Principals perceptions of
internal/external evaluation in the Republic of Ireland and
Northern Ireland. Analysis was carried out using parametric and
non parametric techniques.
Phase 3: - Qualitative Analysis
Interviews with 46 Principals and Inspectors from both regions.
A mixed methods
way of thinking rests
on assumptions that
there are multiple
legitimate
approaches to social
inquiry and that any
given approach to
social inquiry is
inevitably partial
(Greene, 2007 p.20).
Phase 4: - Final analysis (March 2013)
Proposed Framework: for the mutual terms of co-existence
between Internal and External Evaluation.
Centre for Educational Evaluation (CEE), DCU
8. VALUE ADDED ASSESSMENT
(THE GLOBALISED FUTURE)
OECD REVIEW OF EVALUATION AND ASSESSMENT (Australia)
The development of “value-added” models represents significant
progress as they are designed to control for the individual student’s
previous results, and therefore have the potential to identify the
contribution an individual teacher made to
a
student’s
achievement. (OECD p.96, 2011)
OECD REVIEW OF EVALUATION AND ASSESSMENT (Portugal)
In value-added models, students’ actual test scores are often compared
to the projected scores, and classroom and school scores that exceed
the projected values are considered as positive evidence of
instructional effectiveness. (OECD p.132, 2012)
OECD REVIEW OF EVALUATION AND ASSESSMENT
Belgium (Flemish Community), Chile, Czech Republic, Denmark,
Luxembourg, Mexico, Netherlands, New Zealand, Norway, Portugal, Slovak
Republic, Sweden and the United Kingdom (Northern Ireland).
Centre for Educational Evaluation (CEE), DCU
9. VALUE ADDED ASSESSMENT
THE IRISH FUTURE
JUNIOR CYCLE FRAMEWORK
The DES will provide each school with a Data Profile…The
Data Profile will also provide schools with information on their
patterns of achievement relative to schools with a similar
school context…These data will help schools to refine their
assessment and moderation practice. They will also be a
valuable source of information for schools’ self-evaluation
processes…In the event of an unusual pattern of
achievement, the Inspectorate of the DES will be advised, and
support and evaluation measures will be provided for the school
(DESROI, 2012a, p.27).
Centre for Educational Evaluation (CEE), DCU
10. SELF EVALUATION: THE PRESENT
Over a four-year period from 2012, all post-primary
schools should engage in school self-evaluation and
produce three-year improvement plans for numeracy,
literacy and one aspect of teaching and learning
across all subjects and programmes
(Department of Education and Skills 2012, p.2).
The performative culture is so deeply ingrained in
schools and education systems that I can foresee a
game of permanent artifice, where schools squeeze
their individual circumstances into a self-evaluation
document designed solely to impress inspectors,
and hold themselves in a state of perpetual
readiness to live up to their claims, the model
prisoner. In this context, ‘bleak indeed is the
desire for perfection’ (Marshall 1999, 310)’
(Perryman 2009, p.629)
Centre for Educational Evaluation (CEE), DCU
11. FRAMEWORK FOR THE CO-EXISTENCE OF
INTERNAL AND EXTERNAL EVALUATION
AT A CONCEPTUAL LEVEL
Evaluation needs to be perceived as a means of understanding
rather than judgement’ (Nevo 2002, p.10).
AT A METHODOLOGICAL LEVEL
Evaluation should be based on the continuous dissemination
of information, enabling mutual learning. However, Ryan et al
(2007, p.208) ask ‘what should we expect from novice, schoolbased evaluators? Should we have the same standards and
expectations for the school teams (i.e., internal evaluators) as we
do for evaluators conducting external evaluations?’
AT A COMMUNICATION LEVEL
There needs to be mutual respect and trust between internal
and external evaluators. However, within the social fabric of
many regions, such as the ROI, where the economic downturn
has led to cut backs in education, it would be reasonable to
suggest that very little trust in government policy and practice in
general actually exists.
Centre for Educational Evaluation (CEE), DCU
12. FRAMEWORK FOR THE CO-EXISTENCE OF
EVALUATION
AT AN INFLUENTIAL LEVEL
In the past, many Evaluation systems tended towards, Reward, Coercive and Legitimate
power.
Table 2.3:Ravens (1965) typology for the six bases of power
Positional vs transformational bases of power
POWER BASE
POSITIONAL
DEPENDENT ON CHANGE
IN TARGETS INTERNAL
BELIEFS
REQUIRES
SURVEILLANCE
Reward
Yes
No
Yes
Coercive
Yes
No
Yes
Legitimate power
Yes
No
Yes
Referent power
No
Yes
No
Expert power
No
Yes
No
Informational power
No
Yes
No
However, in order for internal and external evaluation to mutually and beneficilally co-exist:
Evaluation frameworks should centre on the Referent, Expert and Informational aspect of
Evaluation.
Centre for Educational Evaluation (CEE), DCU
13. FRAMEWORK FOR THE CO-EXISTENCE OF
INTERNAL AND EXTERNAL EVALUATION
USE OF THE INFLUENTIAL ASPECT OF EVALUATION
During my time as Principal the emphasis was on schools
evaluating themselves with the Inspector as the "critical
friend“ [Expert Power]. For the approach to be successful it
required a good working relationship between the Principal
and the District Inspector… [Referent Power]. By meeting
frequently we grew to trust and respect one another with the
result that I never felt threatened or worried when
[Dr.Shevlin] (DI) would ask probing questions… With [Dr.
Shevlins] support we tried new approaches to things like
literacy, discipline, behaviour and then wrote up our policies
[Informational Power]. It was this professional relationship
between the Inspector and the Principal which greatly
assisted school improvement in the case of St Colm's…
(Hansard Archive, Northern Ireland Assembly, 2011)
Evaluator Influence
(St. Colm’s High School
Belfast)
Centre for Educational Evaluation (CEE), DCU
14. FRAMEWORK FOR THE CO-EXISTENCE OF
INTERNAL AND EXTERNAL EVALUATION
AT A CULTURAL RESPONSIVE LEVEL
I can find no logical explanation as to why our evaluations
should not be culturally responsive or that we should not
behave in culturally responsible ways in our work as
evaluators. (Hood, 2001)
Indeed:
Evaluation efforts have typically failed to consider cultural
background and context in their design, implementation,
analyses, and recommendations.
However:
It is not possible to effectively derive evaluative meaning from
educational programs, designed to serve culturally diverse
students, unless the evaluator themselves are more culturally
responsive. (Hood, 2001)
Centre for Educational Evaluation (CEE), DCU
15. CONCLUSION
EVALUATION
Those of us who are proponents of external evaluation should find ways
to empower schools and teachers to participate as equal partners in the
evaluation process and make use of it; and those of us who believe in
internal evaluation as a means for school autonomy and teacher
professionalisation must admit the legitimacy of accountability and the
right of the public to know. They, in their turn, should seek external
evaluation as a partner for dialog rather than an object for rejection.
(Nevo 2010, p.784)
ASSESSMENT
Every time you use a measure of pupil attainment for some extraneous
purpose you risk creating a perverse incentive. So every time you find
yourself doing that or participating in a system that requires you to do that;
ask what incentive is this creating? Whom is it damaging? Pupils, head
teachers, schools, employers and I think that would be a very useful first
step (O’Neill, 2011).
Centre for Educational Evaluation (CEE), DCU
16. Initial findings: Principals perceptions of External Evaluation
Disagree
Disagree
Indifferent
Strongly
South
Count
9
58
64
%
2.5%
16.0%
17.6%
North
Count
1
17
25
%
1.0%
17.2%
25.3%
U(1) = 16805.500, Z =-1.100 , p = .271
Table 4.4.7: External evaluation results in better teaching and learning
Region
Region
South
Count
%
Disagree
Strongly
16
4.4%
North
Count
1
%
1.0%
U(1) = 17885.500 , Z = -.077 , p =.939
Agree
210
57.9%
52
52.5%
Agree
Strongly
22
6.1%
4
4.0%
Disagree
Indifferent
Agree
57
78
187
Agree
Strongly
25
15.7%
17
17.2%
21.5%
23
23.2%
51.5%
53
53.5%
6.9%
5
5.1%
Table 4.4.3: External evaluation results in better Management
Centre for Educational Evaluation (CEE), DCU
17. Initial findings: Principals perceptions of External Evaluation
Disagree
Disagree
Indifferent
Strongly
South
Count
9
58
64
%
2.5%
16.0%
17.6%
North
Count
1
17
25
%
1.0%
17.2%
25.3%
U(1) = 16805.500, Z =-1.100 , p = .271
Table 4.4.7: External evaluation results in better teaching and learning
Region
Region
South
Count
%
Disagree
Strongly
16
4.4%
North
Count
1
%
1.0%
U(1) = 17885.500 , Z = -.077 , p =.939
Agree
210
57.9%
52
52.5%
Agree
Strongly
22
6.1%
4
4.0%
Disagree
Indifferent
Agree
57
78
187
Agree
Strongly
25
15.7%
17
17.2%
21.5%
23
23.2%
51.5%
53
53.5%
6.9%
5
5.1%
Table 4.4.3: External evaluation results in better Management
Centre for Educational Evaluation (CEE), DCU
