Psychological aspects of CAT:seeking item selection rules whichdo not decrease test takers’ learningself-efficacy and moti...
1CAT for Everyone    MOTTO      &  Happy CAT
2Outline• Previous Study (Kimura & Nagaoka, 2011) ▫ How test-takers feel about CAT• Current Study                      UCA...
3Previous Study: Questionnaire Research(Kimura & Nagaoka, 2011)Participants: 156 Japanese university freshmenOne week afte...
4Previous Study: Questionnaire Research(Kimura & Nagaoka, 2011)Participants: 156 Japanese university freshmenOne week afte...
5Previous Study: Questionnaire Research(Kimura & Nagaoka, 2011)Participants: 156 Japanese university freshmenOne week afte...
6Students reaction to CAT(Kimura & Nagaoka, 2011)                    More difficult      I could not  It was so           ...
7Students reaction to CAT(Kimura & Nagaoka, 2011)  Every students tend to get 50% correct in CAT:  because such items maxi...
8Students reaction to CAT(Kimura & Nagaoka, 2011) School Exams                 60-80%                  40-60%             ...
9 Hypothesis(Kimura & Nagaoka, 2011)CAT tends to be very difficult to most of the students.Experience of taking CAT discou...
10 Conclusion (Kimura & Nagaoka, 2011)Most of the students (90%) feel CAT difficult.Many of them (60%) feel discourage/uns...
11Current Study: Research QuestionWill manipulation of item difficulty in itemselection rules change students reactions to...
12Current Study: DesignAbout 200 Japanese university freshmen took both CAT A& CAT B with one month interval. They also an...
13Current Study: Result1. Did you feel the CAT easy or difficult?                  Very                  A little       Ju...
14Current Study: Result2. Did you feel discouraged or unsatisfied with the   experience?         Very discouraged, 9%     ...
15Current Study: Result2. Did you feel discouraged or unsatisfied with the   experience?         Very discouraged, 9%     ...
16Current Study: Result3. How much do you . . . ?100                                                       Previous 90    ...
17Current Study ConclusionChanging the targeting P of item selection from 0.5 to0.7, with 20% increase in the number of it...
18Current Study: Result3. How much do you . . . ?100                                                       Previous 90    ...
19Thank you for listening.                         Tetsuo Kimura                         (tetsuo.kmr@gmail.com) Acknowledg...
Upcoming SlideShare
Loading in …5
×

Psychological aspects of CAT: seeking item selection rules which do not decrease test takers’ learning self-efficacy and motivation

424 views

Published on

A presentation at IACAT 2012 Sydney

Published in: Lifestyle, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
424
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Psychological aspects of CAT: seeking item selection rules which do not decrease test takers’ learning self-efficacy and motivation

  1. 1. Psychological aspects of CAT:seeking item selection rules whichdo not decrease test takers’ learningself-efficacy and motivationTetsuo KIMURA *1 *2 & Keizo NAGAOKA *2*1 Niigata Seiryo University*2 Graduate School of Human Science, Waseda University IACAT Conference 2012 (Sydney, Australia) Aug. 12-14, 2012
  2. 2. 1CAT for Everyone MOTTO & Happy CAT
  3. 3. 2Outline• Previous Study (Kimura & Nagaoka, 2011) ▫ How test-takers feel about CAT• Current Study UCAT ▫ Research Question ▫ Designe ▫ Result ▫ Conclusion
  4. 4. 3Previous Study: Questionnaire Research(Kimura & Nagaoka, 2011)Participants: 156 Japanese university freshmenOne week after they had taken an English CAT (CASEC)1. Did you feel the CAT easy or difficult? Just right, 13 Very A little A little easy,1 difficult, 82 difficult, 60 Very easy, 00% 20% 40% 60% 80% 100%
  5. 5. 4Previous Study: Questionnaire Research(Kimura & Nagaoka, 2011)Participants: 156 Japanese university freshmenOne week after they had taken an English CAT (CASEC)2. Did you feel discouraged or unsatisfied with the experience? Dicouraged, Neutral, 35 Satisfied, 26 950% 20% 40% 60% 80% 100%
  6. 6. 5Previous Study: Questionnaire Research(Kimura & Nagaoka, 2011)Participants: 156 Japanese university freshmenOne week after they had taken an English CAT (CASEC)3. How much do you . . . ? 100 80 76 66 70 60 % 45 40 34 20 0 High School CASEC Satisfied Expected Disapointed
  7. 7. 6Students reaction to CAT(Kimura & Nagaoka, 2011) More difficult I could not It was so than entrance answer with difficult  exam  confidence  Hana Yoko Taro I may not beI dislike English Worst score I’ve able to reach theworse  ever got  passing grade 
  8. 8. 7Students reaction to CAT(Kimura & Nagaoka, 2011) Every students tend to get 50% correct in CAT: because such items maximize test information for each student. However, they feel 34% correct. Hana Yoko Taro False Low Advanced beginner Intermediate Intermediate
  9. 9. 8Students reaction to CAT(Kimura & Nagaoka, 2011) School Exams 60-80% 40-60% 80-100% Hana Yoko Taro Entrance Exams 60
  10. 10. 9 Hypothesis(Kimura & Nagaoka, 2011)CAT tends to be very difficult to most of the students.Experience of taking CAT discourages students.Backwash effects Loss of self-confidence Loss of self-efficacy Loss of motivation
  11. 11. 10 Conclusion (Kimura & Nagaoka, 2011)Most of the students (90%) feel CAT difficult.Many of them (60%) feel discourage/unsatisfied withthe experience.More attention may have to be paid to thepsychological effects of CAT.Item selection rule to maximize information shouldbe changed even if it makes the test a little longer.
  12. 12. 11Current Study: Research QuestionWill manipulation of item difficulty in itemselection rules change students reactions to CAT? Minimum number of CAT Items AdministeredTargetin S.E. (logits) gP 0.5 0.4 0.3 0.2 0.1 0.5 16 25 45 100 499 CAT A 0.6 17 27 47 105 517 0.7 20 30 53 120 477 CAT B 0.8 25 40 70 157 625 0.9 45 70 125 278 1112 Linacre (2006)
  13. 13. 12Current Study: DesignAbout 200 Japanese university freshmen took both CAT A& CAT B with one month interval. They also answeredquestionnaire immediately after each CAT Targeting Number Expected P of Item S.E. Vocabulary & 0.5 16 0.5 GrammarCAT A Listening 0.5 16 0.5 Comprehension Vocabulary & 0.7 20 0.5 GrammarCAT B Listening 0.7 20 0.5 Comprehension
  14. 14. 13Current Study: Result1. Did you feel the CAT easy or difficult? Very A little Just right,CAT B difficult, 36% difficult, 49% 13% A little easy, 1% Very A little Just right,CAT A difficult, 49% difficult, 45% 6% A little easy, 0% 0% 20% 40% 60% 80% 100% χ2 = 11.8 p = 0.008
  15. 15. 14Current Study: Result2. Did you feel discouraged or unsatisfied with the experience? Very discouraged, 9% A littele satisfied, 19%CAT B Neutral, 51% A littele Very satisfied, discouraged, 14% 8% Very discouraged, 13% A littele satisfied, 10%CAT A A littele Neutral, 47% Very satisfied, discouraged, 22% 7% 0% 20% 40% 60% 80% 100%
  16. 16. 15Current Study: Result2. Did you feel discouraged or unsatisfied with the experience? Very discouraged, 9% A littele satisfied,CAT B 23% A littele Neutral, 51% 27% 19% Very satisfied, discouraged, 14% 8% Very discouraged, 13% A littele satisfied, 10%CAT A 35% A littele discouraged, 22% Neutral, 47% 17% Very satisfied, 7% 0% 20% 40% 60% 80% 100% χ2 = 12.4 p = 0.014
  17. 17. 16Current Study: Result3. How much do you . . . ?100 Previous 90 CAT A 80 CAT B 70 60 50 40 30 20 10 0 High School CAT Satisfied Expected Disapointed
  18. 18. 17Current Study ConclusionChanging the targeting P of item selection from 0.5 to0.7, with 20% increase in the number of itemLess test-takers feel the CAT very difficultMore test-takers feel the CAT just rightLess test-takers feel discouraged with the experienceMore test-takers feel satisfied with the experienceThis item selection rule less decreases test takers’learning self-efficacy and motivation
  19. 19. 18Current Study: Result3. How much do you . . . ?100 Previous 90 CAT A 80 CAT B 70 60 50 40 30 20 10 0 High School CAT Satisfied Expected Disapointed
  20. 20. 19Thank you for listening. Tetsuo Kimura (tetsuo.kmr@gmail.com) Acknowledgements: A part of the present study was supported by a Grant-in- Aid for Scientific Research for 2010-2012 (No. 22520590) from the Japan Society for the Promotion of Science.

×