International Literacy Assessment in Indonesia ￼￼￼￼￼- Suhendra Yusuf
International Literacy Assessment in Indonesia Suhendra Yusuf Bincang Edukasi Jakarta, 19 Februari 2013 1
Overview l During the ﬁrst decade of the Third Millennium, Indonesia has taken part in several international studies on students’ achievement i.e., PIRLS, PISA, TIMSS, and IBT as an eﬀort to map its educational standards compared to the global ones. l The results were still not yet satisfactory. The students’ average score in PIRLS 2006 was 407 and PIRLS 2011 was 428; all was below international average score 500. l PISA 2000-‐2009 showed that the highest score in Reading literacy was 402 (2009), Math 391 (2006), and Science 395 (2003/6). l TIMSS 1999-‐2011 also portrayed the highest score in Math (Grade 8) was 411 (2003) and Science 435 (1999). l However, if the sample is focused on limited numbers of students in international schools, IBT (2009) showed the highest score in Math (Grade 10) was 753 and the lowest 523, average score 607. l Results of international assessments have grown issues of discussion among the Ministry of Education, the House of Representatives, and experts in universities. Results have prompted a review of education policy. 2
Reasons to participate in international assessments l The main reasons to participate in international assessments are to monitor of quality of education and to describe and understand determinant factors and observed diﬀerences of the students achievement among provinces in Indonesia. l Program was proposed by the Center of National Education Assessment (CNEA/Puspendik) but the decision to participate was made by the National Education Research and Development Body, the Ministry of Education and Culture MoEC). l The CNEA implemented the Assessments. Experts from higher institutions were involved in the preparation, implementation, monitoring and evaluation of the program. l Pros and cons on the assessment participation. The opponent arguments: § Sample was too small. It did not reﬂect the whole population. § Test items and questionnaires were out of context. § Translated materials were not suﬃcient or inappropriate. § Assessment results only exacerbate or worsen the image of educational achievement 3 internationally.
Managing the Results l Results were analyzed and discussed among experts in several academic meetings facilitated by the Center of National Education Assessment. l Seminars were held to discuss the results in several occasions involving all educational stakeholders. Results were recommended to policy makers. l Results were also exposed to policy makers in national, provincial, and local levels. l The MoEC held a hearing with the House of Representatives to discuss aspects of educational policy based on critical educational issues. l Exposure to policy makers in the House of Representatives, national, provincial, and local authorities in education had the most impact. l Results were also published as a reference book. 4
Improving the education system l Results of international assessments have grown awareness and issue of discussion among the MoEC authorities and the Indonesia House of Representatives. § Quoting results in formal and informal meetings § Becoming issues in mass media l Results have prompted a review of education policy. § Curriculum 2013 was among other inﬂuenced by the international assessment results e.g., integrated subjects and more time to read in elementary schools. § Data from previous international assessments were usually compared and analyzed. l The CNEA developed a program similar to international assessment called Indonesian National Assessment Program (INAP) to improve the quality of education. § Using Indonesian and local contexts § Referring to content standards in the curriculum 5
The Challengesl The existing curriculum: § Reading literacy is the bases for Math and Science literacy development § Reading Curriculum has not been changed since 1994. It is not a separate curriculum area. There are only language teachers, not reading teachers § Instructional time devoted to language/Reading Curriculum was the lowest (15%) among countries § Curriculum has no emphasis on reading engagement/ reading for enjoyment § Reading curriculum is not focused on certain cognitive processes and does not include the highest cognitive process, i.e., examining/evaluating content, language and textual elements § Methods used to evaluate reading curriculum is not based on research § Curriculum for secondary school does not include Chemistry. Only 32% of TIMSS sample said they have chemistry subject, resulting in the lowest among science strand § Geography is not part of science strand; it is a social science strand § Less contextual and rational thinking process in math curriculum 6
The Challengesl Teacher preparation is not yet suﬃcient § Most teachers receive their education through a teacher college program § They are supervised during education and have to pass qualifying exam § No completion of probationary teaching period § No completion of mentoring or induction program l ESCS (Economic-‐Social-‐Cultural Status) Index is the lowest among participating countries: § The increase 1 point in the ESCS (parent education/occupation, home possession / number of books at home index) will improve 17 points in students performance § ESCS inﬂuences students’ reading engagement: enjoyment in reading contributes 43% and diversity of reading materials inﬂuences 60% to students performance § ESCS inﬂuences also students learning strategy: memorization and elaboration strategies contribute 34% and 25% respectively to students performance 7
Reading Curriculum has not beenchanged since 1994
Lowest Instructional Time in Reading Curriculum
Indonesia Thailand Tunisia Brazil Albania Hong Kong-‐China Indonesia lowest index does not result in lowest achievement… Uruguay Azerbaijan Trinidad and Tobago Jordan Shanghai-‐China Romania Portugal Poland Russian Federation Croatia Lowest ESCS Index (PISA 2009) France Italy Slovak Republic Lithuania Greece Ireland Serbia Switzerland New Zealand United States Luxembourg United Kingdom Denmark Australia Dubai (UAE) Canada Iceland 1.00 0.50 0.00 -0.50 -1.00 -1.50 -2.00
Reading Performance and Social-‐economic Background (PISA 2009) SCORE > OECD 600 SCORE > OECD ESCS < OECD China, Singapore, Japan, Korea ESCS > OECD 500 § Thailand § Indonesia 400 Y = 56X ; R2 = 0,32 § Peru § Qatar In spite of ESCS lowest index, 300 Indonesia has better achievement compared to higher indexes e.g. 200 Peru and Qatar SCORE < OECD 100 SCORE < OECD ESCS INDEX < OCD ESCS > OECD 0-2 -1.5 -1 -0.5 0 0.5 1 1.5
Detailed Social-‐economic Factors Inﬂuencing Reading Performance (PISA 2009) The increase 1 point in the ESCS index will improve 17 points in students’ reading performance
Engagement in Reading, Learning Strategies & Reading Performance (PISA 2009) Diversity of reading materials and memorizing strategy mostly inﬂuence reading performance
Factors Inﬂuencing School Background (PISA 2009) School resources, school size, and teachers quality inﬂuence signiﬁcantly school social-‐economy index which has impact on the students performance.
The Strengths…l The National Examination held by the Ministry of Education and Culture (2009-‐2011) for Grade 9 showed a signiﬁcant improvement in Science and English. In spite of a decrease in Math and Indonesian, the scores are above international achievement. o Test items with local contexts get better results than those with international contexts. l The International Benchmark Test (IBT) – using limited numbers of students in international schools as the sample of the study – showed the highest score in Math (Grade 10) was 753 and the lowest 523, average score 607. o There are 1300 schools of this category (out of 182.538 schools with 58 million students) all over the country. 19
International Benchmark Test (IBT)l IBT is designed by ACER (Australian Council for Educational Research) to assess the performance of students from school against local, national and international standards. o The tests are in English, Math, and Science. o The Math strands are chance and data, measurement, number, and space. o The English strands are comprehension, punctuation, spelling, grammar, and vocabulary. o The Science strands are physical science, earth science, and life science. l Student results for the IBT are compared with TIMSS 2007 for: o Mathematics in Years 4, 5, 8 and 9 o Science in Years 4, 5 and 8 21
IBT Results No. Subjects Test Schools Grade No. of Mean Score range Level students score 1. Math 10 SMAN 8 Pekanbaru 10 242 607 523 – 735 2. Math 9 SMAN 1 Mataram 10 212 614 419 – 749 3. Math 10 SMAN 1 Mataram 11 61 618 513 – 687 4. Math 10 SMAN 1 Mataram 12 48 647 559 – 759 Total 621 419 – 759 5. English 10 SMAN 8 Pekanbaru 10 242 525 422 – 627 6. English 10 SMAN 1 Mataram 10 211 505 319 – 602 7. English 10 SMAN 1 Mataram 11 61 562 499 – 643 8. English 10 SMAN 1 Mataram 12 46 557 490 – 661 Total 537 319 – 661 22
Lesson Learned… l International assessments have been used to monitor and compare the quality of education locally and internationally and to understand factors inﬂuencing the students achievement. l The program has become issues of discussion among the MoEC authorities, the House of Representatives, and academician. Results have been used to review the education policy. l The opponents to program participation particularly argue on the sample representativeness, the quality of test materials, and the unexpected results to worsen the image of quality of education achievement internationally.
Lesson Learned… l The need to reform the curriculum: § Results of international-‐level studies might be accounted for by diﬀerences in curriculum rather than intellectual diﬀerences among students (Gustafsson and Undheim, 1996) § Reading has to be a separate curriculum area. There should be more reading teachers § Instructional time devoted to reading has to be increased at least twice from current curriculum § Curriculum has more emphasis on reading for enjoyment § Reading curriculum has to focus on certain cognitive processes, to include process of examining/evaluating content, language and textual elements § More research in implementation of reading curriculum has to be conducted
Lesson Learned… l Reading literacy is the basis of Math and Science literacy development. o There is a need to a more comprehensive approach to reading instruction to include teaching for motivation. o The need of professional development for subject teachers, including training in the use of research-‐based reading instruction . o The need of classroom-‐based strategies for improving student reading and comprehension strategies across the curriculum. o More professional initial and continuing teacher education. The International Reading Association (1998) recommended that primary teachers have 280 hours of instruction in reading and how to teach it.
Lesson Learned…l There is a need to continuing teacher education and appropriate ongoing professional development in reading, math, and science. l There should be an eﬀective intervention for children experiencing diﬃculties. l As of the sample of the next study: better to focus purposively on a certain type of schools as a benchmark or a model of school development.
Thank you very much Terima kasih Suhendra Yusuf firstname.lastname@example.org 29
Dr. Suhendra Yusuf o Project Director International Benchmark Test (IBT), Institut Asesmen Indonesia (IAI) in collaboration with Australia Council for Educational Research Australia o International Quality Control Monitor for the implementation of PIRLS 2006 & 2011 o Researcher PISA & PIRLS 2000 – 2011 o Associate Professor and vice Rector Nusantara Islamic University Bandung o Author Benchmark Internasional Mutu Pendidikan (co-‐ authored with Bahrul Hayat, Bumi Aksara, 2010)