1. International
Literacy
Assessment
in
Indonesia
Suhendra
Yusuf
Bincang
Edukasi
Jakarta,
19
Februari
2013
1

2. Overview
l During
the
first
decade
of
the
Third
Millennium,
Indonesia
has
taken
part
in
several
international
studies
on
students’
achievement
i.e.,
PIRLS,
PISA,
TIMSS,
and
IBT
as
an
effort
to
map
its
educational
standards
compared
to
the
global
ones.
l The
results
were
still
not
yet
satisfactory.
The
students’
average
score
in
PIRLS
2006
was
407
and
PIRLS
2011
was
428;
all
was
below
international
average
score
500.
l PISA
2000-­‐2009
showed
that
the
highest
score
in
Reading
literacy
was
402
(2009),
Math
391
(2006),
and
Science
395
(2003/6).
l TIMSS
1999-­‐2011
also
portrayed
the
highest
score
in
Math
(Grade
8)
was
411
(2003)
and
Science
435
(1999).
l However,
if
the
sample
is
focused
on
limited
numbers
of
students
in
international
schools,
IBT
(2009)
showed
the
highest
score
in
Math
(Grade
10)
was
753
and
the
lowest
523,
average
score
607.
l Results
of
international
assessments
have
grown
issues
of
discussion
among
the
Ministry
of
Education,
the
House
of
Representatives,
and
experts
in
universities.
Results
have
prompted
a
review
of
education
policy.
2

3. Reasons
to
participate
in
international
assessments
l The
main
reasons
to
participate
in
international
assessments
are
to
monitor
of
quality
of
education
and
to
describe
and
understand
determinant
factors
and
observed
differences
of
the
students
achievement
among
provinces
in
Indonesia.
l Program
was
proposed
by
the
Center
of
National
Education
Assessment
(CNEA/Puspendik)
but
the
decision
to
participate
was
made
by
the
National
Education
Research
and
Development
Body,
the
Ministry
of
Education
and
Culture
MoEC).
l The
CNEA
implemented
the
Assessments.
Experts
from
higher
institutions
were
involved
in
the
preparation,
implementation,
monitoring
and
evaluation
of
the
program.
l Pros
and
cons
on
the
assessment
participation.
The
opponent
arguments:
§ Sample
was
too
small.
It
did
not
reflect
the
whole
population.
§ Test
items
and
questionnaires
were
out
of
context.
§ Translated
materials
were
not
sufficient
or
inappropriate.
§ Assessment
results
only
exacerbate
or
worsen
the
image
of
educational
achievement
3
internationally.

4. Managing
the
Results
l Results
were
analyzed
and
discussed
among
experts
in
several
academic
meetings
facilitated
by
the
Center
of
National
Education
Assessment.
l Seminars
were
held
to
discuss
the
results
in
several
occasions
involving
all
educational
stakeholders.
Results
were
recommended
to
policy
makers.
l Results
were
also
exposed
to
policy
makers
in
national,
provincial,
and
local
levels.
l The
MoEC
held
a
hearing
with
the
House
of
Representatives
to
discuss
aspects
of
educational
policy
based
on
critical
educational
issues.
l Exposure
to
policy
makers
in
the
House
of
Representatives,
national,
provincial,
and
local
authorities
in
education
had
the
most
impact.
l Results
were
also
published
as
a
reference
book.
4

5. Improving
the
education
system
l Results
of
international
assessments
have
grown
awareness
and
issue
of
discussion
among
the
MoEC
authorities
and
the
Indonesia
House
of
Representatives.
§ Quoting
results
in
formal
and
informal
meetings
§ Becoming
issues
in
mass
media
l Results
have
prompted
a
review
of
education
policy.
§ Curriculum
2013
was
among
other
influenced
by
the
international
assessment
results
e.g.,
integrated
subjects
and
more
time
to
read
in
elementary
schools.
§ Data
from
previous
international
assessments
were
usually
compared
and
analyzed.
l The
CNEA
developed
a
program
similar
to
international
assessment
called
Indonesian
National
Assessment
Program
(INAP)
to
improve
the
quality
of
education.
§ Using
Indonesian
and
local
contexts
§ Referring
to
content
standards
in
the
curriculum
5

6. The Challenges
l The
existing
curriculum:
§ Reading
literacy
is
the
bases
for
Math
and
Science
literacy
development
§ Reading
Curriculum
has
not
been
changed
since
1994.
It
is
not
a
separate
curriculum
area.
There
are
only
language
teachers,
not
reading
teachers
§ Instructional
time
devoted
to
language/Reading
Curriculum
was
the
lowest
(15%)
among
countries
§ Curriculum
has
no
emphasis
on
reading
engagement/
reading
for
enjoyment
§ Reading
curriculum
is
not
focused
on
certain
cognitive
processes
and
does
not
include
the
highest
cognitive
process,
i.e.,
examining/evaluating
content,
language
and
textual
elements
§ Methods
used
to
evaluate
reading
curriculum
is
not
based
on
research
§ Curriculum
for
secondary
school
does
not
include
Chemistry.
Only
32%
of
TIMSS
sample
said
they
have
chemistry
subject,
resulting
in
the
lowest
among
science
strand
§ Geography
is
not
part
of
science
strand;
it
is
a
social
science
strand
§ Less
contextual
and
rational
thinking
process
in
math
curriculum
6

7. The Challenges
l Teacher
preparation
is
not
yet
sufficient
§ Most
teachers
receive
their
education
through
a
teacher
college
program
§ They
are
supervised
during
education
and
have
to
pass
qualifying
exam
§ No
completion
of
probationary
teaching
period
§ No
completion
of
mentoring
or
induction
program
l ESCS
(Economic-­‐Social-­‐Cultural
Status)
Index
is
the
lowest
among
participating
countries:
§ The
increase
1
point
in
the
ESCS
(parent
education/occupation,
home
possession
/
number
of
books
at
home
index)
will
improve
17
points
in
students
performance
§ ESCS
influences
students’
reading
engagement:
enjoyment
in
reading
contributes
43%
and
diversity
of
reading
materials
influences
60%
to
students
performance
§ ESCS
influences
also
students
learning
strategy:
memorization
and
elaboration
strategies
contribute
34%
and
25%
respectively
to
students
performance
7

8. Reading Curriculum has not been
changed since 1994

9. Lowest Instructional Time in Reading Curriculum

10. No
emphasis
on
reading
engagement

11. Not
focused
on
certain
cogni3ve
processes

12. Evaluation is not based on research

13. Teacher preparation is not yet sufficient

14. Indonesia
Thailand
Tunisia
Brazil
Albania
Hong
Kong-­‐China
Indonesia
lowest
index
does
not
result
in
lowest
achievement…
Uruguay
Azerbaijan
Trinidad
and
Tobago
Jordan
Shanghai-­‐China
Romania
Portugal
Poland
Russian
Federation
Croatia
Lowest
ESCS
Index
(PISA
2009)
France
Italy
Slovak
Republic
Lithuania
Greece
Ireland
Serbia
Switzerland
New
Zealand
United
States
Luxembourg
United
Kingdom
Denmark
Australia
Dubai
(UAE)
Canada
Iceland
1.00
0.50
0.00
-0.50
-1.00
-1.50
-2.00

15. Reading
Performance
and
Social-­‐economic
Background
(PISA
2009)
SCORE > OECD
600 SCORE > OECD
ESCS < OECD China,
Singapore,
Japan,
Korea
ESCS > OECD
500
§ Thailand
§ Indonesia
400 Y = 56X ; R2 = 0,32
§ Peru
§ Qatar
In
spite
of
ESCS
lowest
index,
300
Indonesia
has
better
achievement
compared
to
higher
indexes
e.g.
200
Peru
and
Qatar
SCORE < OECD 100 SCORE < OECD
ESCS INDEX < OCD ESCS > OECD
0
-2 -1.5 -1 -0.5 0 0.5 1 1.5

16. Detailed
Social-­‐economic
Factors
Influencing
Reading
Performance
(PISA
2009)
The
increase
1
point
in
the
ESCS
index
will
improve
17
points
in
students’
reading
performance

17. Engagement
in
Reading,
Learning
Strategies
&
Reading
Performance
(PISA
2009)
Diversity
of
reading
materials
and
memorizing
strategy
mostly
influence
reading
performance

18. Factors
Influencing
School
Background
(PISA
2009)
School
resources,
school
size,
and
teachers
quality
influence
significantly
school
social-­‐
economy
index
which
has
impact
on
the
students
performance.

19. The Strengths…
l The
National
Examination
held
by
the
Ministry
of
Education
and
Culture
(2009-­‐2011)
for
Grade
9
showed
a
significant
improvement
in
Science
and
English.
In
spite
of
a
decrease
in
Math
and
Indonesian,
the
scores
are
above
international
achievement.
o Test
items
with
local
contexts
get
better
results
than
those
with
international
contexts.
l The
International
Benchmark
Test
(IBT)
–
using
limited
numbers
of
students
in
international
schools
as
the
sample
of
the
study
–
showed
the
highest
score
in
Math
(Grade
10)
was
753
and
the
lowest
523,
average
score
607.
o There
are
1300
schools
of
this
category
(out
of
182.538
schools
with
58
million
students)
all
over
the
country.
19

20. National
Exam
(Grade-­‐9)
Subject
2009
2010
2011
score
sd
score
sd
score
sd
Mathematics
7.60
1.57
7.53
1.28
7.3
1.70
Science
7.32
1.28
7.32
1.17
7.41
1.35
Indonesian
7.38
1.19
7.47
1.08
7.12
1.18
English
7.14
1.45
7.14
1.25
7.52
1.50
20

21. International
Benchmark
Test
(IBT)
l IBT
is
designed
by
ACER
(Australian
Council
for
Educational
Research)
to
assess
the
performance
of
students
from
school
against
local,
national
and
international
standards.
o The
tests
are
in
English,
Math,
and
Science.
o The
Math
strands
are
chance
and
data,
measurement,
number,
and
space.
o The
English
strands
are
comprehension,
punctuation,
spelling,
grammar,
and
vocabulary.
o The
Science
strands
are
physical
science,
earth
science,
and
life
science.
l Student
results
for
the
IBT
are
compared
with
TIMSS
2007
for:
o
Mathematics
in
Years
4,
5,
8
and
9
o
Science
in
Years
4,
5
and
8
21

22. IBT
Results
No.
Subjects
Test
Schools
Grade
No.
of
Mean
Score
range
Level
students
score
1.
Math
10
SMAN
8
Pekanbaru
10
242
607
523
–
735
2.
Math
9
SMAN
1
Mataram
10
212
614
419
–
749
3.
Math
10
SMAN
1
Mataram
11
61
618
513
–
687
4.
Math
10
SMAN
1
Mataram
12
48
647
559
–
759
Total
621
419
–
759
5.
English
10
SMAN
8
Pekanbaru
10
242
525
422
–
627
6.
English
10
SMAN
1
Mataram
10
211
505
319
–
602
7.
English
10
SMAN
1
Mataram
11
61
562
499
–
643
8.
English
10
SMAN
1
Mataram
12
46
557
490
–
661
Total
537
319
–
661
22

23. IBT
Results
23

24. IBT
Results
24

25. Lesson
Learned…
l International
assessments
have
been
used
to
monitor
and
compare
the
quality
of
education
locally
and
internationally
and
to
understand
factors
influencing
the
students
achievement.
l The
program
has
become
issues
of
discussion
among
the
MoEC
authorities,
the
House
of
Representatives,
and
academician.
Results
have
been
used
to
review
the
education
policy.
l The
opponents
to
program
participation
particularly
argue
on
the
sample
representativeness,
the
quality
of
test
materials,
and
the
unexpected
results
to
worsen
the
image
of
quality
of
education
achievement
internationally.

26. Lesson
Learned…
l The
need
to
reform
the
curriculum:
§ Results
of
international-­‐level
studies
might
be
accounted
for
by
differences
in
curriculum
rather
than
intellectual
differences
among
students
(Gustafsson
and
Undheim,
1996)
§ Reading
has
to
be
a
separate
curriculum
area.
There
should
be
more
reading
teachers
§ Instructional
time
devoted
to
reading
has
to
be
increased
at
least
twice
from
current
curriculum
§ Curriculum
has
more
emphasis
on
reading
for
enjoyment
§ Reading
curriculum
has
to
focus
on
certain
cognitive
processes,
to
include
process
of
examining/evaluating
content,
language
and
textual
elements
§ More
research
in
implementation
of
reading
curriculum
has
to
be
conducted

27. Lesson
Learned…
l Reading
literacy
is
the
basis
of
Math
and
Science
literacy
development.
o There
is
a
need
to
a
more
comprehensive
approach
to
reading
instruction
to
include
teaching
for
motivation.
o The
need
of
professional
development
for
subject
teachers,
including
training
in
the
use
of
research-­‐based
reading
instruction
.
o The
need
of
classroom-­‐based
strategies
for
improving
student
reading
and
comprehension
strategies
across
the
curriculum.
o More
professional
initial
and
continuing
teacher
education.
The
International
Reading
Association
(1998)
recommended
that
primary
teachers
have
280
hours
of
instruction
in
reading
and
how
to
teach
it.

28. Lesson
Learned…
l There
is
a
need
to
continuing
teacher
education
and
appropriate
ongoing
professional
development
in
reading,
math,
and
science.
l There
should
be
an
effective
intervention
for
children
experiencing
difficulties.
l As
of
the
sample
of
the
next
study:
better
to
focus
purposively
on
a
certain
type
of
schools
as
a
benchmark
or
a
model
of
school
development.

29. Thank
you
very
much
Terima
kasih
Suhendra
Yusuf
suhendrayusuf@gmail.com
29

30. Dr.
Suhendra
Yusuf
o Project
Director
International
Benchmark
Test
(IBT),
Institut
Asesmen
Indonesia
(IAI)
in
collaboration
with
Australia
Council
for
Educational
Research
Australia
o International
Quality
Control
Monitor
for
the
implementation
of
PIRLS
2006
&
2011
o Researcher
PISA
&
PIRLS
2000
–
2011
o Associate
Professor
and
vice
Rector
Nusantara
Islamic
University
Bandung
o Author
Benchmark
Internasional
Mutu
Pendidikan
(co-­‐
authored
with
Bahrul
Hayat,
Bumi
Aksara,
2010)

31. Our
students
are
at
risk..!