Helping Teachers Meet Learner NeedsThrough Innovative Online Diagnostic            Assessment  Jim Lantolf, Matt Poehner, ...
Outline of Presentation• 1. Introduction to Dynamic Assessment: Ableeva• 2. Online Test Logic: Lu• 3. Demonstration: Lu &A...
Computerized dynamic assessment (CDA)• Interventionist (standardized, scripted menu of  mediating prompts)• Mediation deli...
Test design• Each test item focuses on one or two specific  language constructs, e.g.,  – Phonetics, vocabulary, grammar, ...
Test Design• Each item includes four increasingly explicit prompts  – Prompt 1: drawing learners’ attention to the most   ...
Sample item 1: Chinese listeningScript 男:您还有什么事?我马上要出去。 女:那好,王厂长,改天我再来拜访你。What does the man want to do?A.   make a phone c...
Areas of difficulty for sample item 1Script 男:您还有什么事?我马上要出去。 女:那好,王厂长,改天我再来拜访你。• Areas of difficulty revealed from DA sess...
Sample item 1: Presentation in the test                                          8                                        ...
Sample item 1: Prompt 1                          9
Sample item 1: Prompt 2                          10
Sample item 1: Prompt 3                          11
Sample item 1: Prompt 4 and explanation                                          12
Sample item 2: French reading• Language constructs  – Grammar (primary): Past aspect (imperfective), negation  – Discourse...
Sample item 2: Presentation in the test                                          14
Sample item 2: Prompt 1                          15
Sample item 2: Prompt 2                          16
Sample item 2: Prompt 3                          17
Sample item 2: Prompt 4                          18
Sample item 2: ExplanationAlthough the author liked all aspects of the weekends at his grandparentshouse, his enjoyed the ...
Individual Score on French Listening Assessment• Mediated (i.e., prompted) score and Actual (i.e.,  initial attempt to ans...
Performance of Individual Student on French               Listening Assessment• X’s indicate level of prompts required by ...
Performance of Individual Student on French               Listening Assessment• We report the time it takes for a test tak...
Group Performance on French Listening                    Assessment•   Group scores are reported for each item by the leve...
Group Performance on French Listening Assessment•   Scores are also reported by rubric. Thus, the first 5 items included i...
French Reading: Low Intermediate Group        N       Mean           SD     Min   Max   FRA        23     47.478       11....
French Reading: High Intermediate  Group         N        Mean           SD     Min   Max   FRA          42      40.762   ...
French Listening: High Intermediate  Group        N            Mean         SD      Min   Max   FLA         34           2...
Chinese Reading: High Intermediate  Group          N        Mean            SD     Min   Max   CRA           81       47.6...
Chinese Listening: Low Intermediate  Group           N       Mean           SD      Min   Max   CLA           71      42.5...
Learning Potential Scores• “…the learning potential score (LPS) has to reflect  both gain made by the student from pre- to...
Learning Potential Scores: Chinese Reading                    Test                                 LPS1.40001.20001.00000....
LPS for French Listening Chinese Reading                AssessmentACTUAL SCORE   MEDIATED SCORE        LPS     20         ...
Interpretation of LPS• In the previous slide three students had the same ACTUAL score on the Chinese  Reading Assessment: ...
Tests Available Soon thru CALPER• General information about dynamic  assessment, including working paper and  bibliography...
Upcoming SlideShare
Loading in …5
×

Helping Teachers Meet Learner Needs Through Innovative Online Diagnostic Assessment

1,534 views

Published on

Presentation at the 2011 American Council of the Teaching of Foreign Languages (ACTFL). Description of Computerized Dynamic Assessment Tests developed for assessing listening comprehension in Chinese, Russian, and French. Test developed by the Center for Language Acquisition (CLA) and the Center for Advanced Language Proficiency Education and Research (CALPER) at the Pennsylvania State University.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,534
On SlideShare
0
From Embeds
0
Number of Embeds
433
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Helping Teachers Meet Learner Needs Through Innovative Online Diagnostic Assessment

  1. 1. Helping Teachers Meet Learner NeedsThrough Innovative Online Diagnostic Assessment Jim Lantolf, Matt Poehner, Xiaofei Lu, Penn State University RumiaAbleeva, University of Georgia The project was funded by a grant from the U.S. Department of Education: PO17A080071 ACTFL Annual Conference. Denver. November 19, 2011
  2. 2. Outline of Presentation• 1. Introduction to Dynamic Assessment: Ableeva• 2. Online Test Logic: Lu• 3. Demonstration: Lu &Ableeva – Chinese Listening Test – French Reading Test• 4. Scoring Procedures & Learner Profile: Lantolf• 5. Audience Questions & Discussion
  3. 3. Computerized dynamic assessment (CDA)• Interventionist (standardized, scripted menu of mediating prompts)• Mediation delivered via computer program rather than human mediator• Context of computer-based test of L2 listening and reading comprehension in Chinese, French & Russian 3
  4. 4. Test design• Each test item focuses on one or two specific language constructs, e.g., – Phonetics, vocabulary, grammar, discourse, pragmatics• Design of the mediating prompts guided by findings from the the interactive DA sessions 4
  5. 5. Test Design• Each item includes four increasingly explicit prompts – Prompt 1: drawing learners’ attention to the most relevant segments – Prompt 2: drawing learner’s attention to the key language constructs of the item resided in those segments – Prompt 3: offering an explanation for the key language constructs – Prompt 4: offering the correct answer to the question – Each item also contains an explanation for the answer 5
  6. 6. Sample item 1: Chinese listeningScript 男:您还有什么事?我马上要出去。 女:那好,王厂长,改天我再来拜访你。What does the man want to do?A. make a phone callB. visit the womanC. go out for dinnerD. cook at homeE. end the conversation 6
  7. 7. Areas of difficulty for sample item 1Script 男:您还有什么事?我马上要出去。 女:那好,王厂长,改天我再来拜访你。• Areas of difficulty revealed from DA sessions – 出去 ‘go out’: students knew this word but failed to recognize it in listening – 改天 ‘another day’: students knew the characters but not the meaning of the compound word 7
  8. 8. Sample item 1: Presentation in the test 8 8
  9. 9. Sample item 1: Prompt 1 9
  10. 10. Sample item 1: Prompt 2 10
  11. 11. Sample item 1: Prompt 3 11
  12. 12. Sample item 1: Prompt 4 and explanation 12
  13. 13. Sample item 2: French reading• Language constructs – Grammar (primary): Past aspect (imperfective), negation – Discourse (Secondary): Relationship between a negative clause and a mais plutot clause 13
  14. 14. Sample item 2: Presentation in the test 14
  15. 15. Sample item 2: Prompt 1 15
  16. 16. Sample item 2: Prompt 2 16
  17. 17. Sample item 2: Prompt 3 17
  18. 18. Sample item 2: Prompt 4 18
  19. 19. Sample item 2: ExplanationAlthough the author liked all aspects of the weekends at his grandparentshouse, his enjoyed the quiet evenings with his grandfather the most. The keyis the passage that starts with "Ce que jaimais le plus" (what I liked themost). The author states "ce nétait pas les les animaux" (it wasnt theanimals), "mais plutôt les soirées calmes avec mon grand-père" (but ratherthe quiet evenings with my grandfather). He sets up an oppositions betweenthe negative "ce nétait pas les les animaux" with the "les soirées calmes avecmon grand-père" by using the phrase "mais plutôt" (but rather). 19
  20. 20. Individual Score on French Listening Assessment• Mediated (i.e., prompted) score and Actual (i.e., initial attempt to answer prior to prompting)• NB: Maximum score possible on test is 60 (i.e., 4 x 15 items)
  21. 21. Performance of Individual Student on French Listening Assessment• X’s indicate level of prompts required by item. Highest score on each item = 4 (i.e., initial attempt to respond before prompts are provided)
  22. 22. Performance of Individual Student on French Listening Assessment• We report the time it takes for a test taker to initially respond to each item and the time it takes for the individual to respond following each required prompt. – It took the individual here 23 seconds to provide an initial response and 5 seconds to respond to the first prompt. The individual provided the correct response (a) following the first prompt and therefore earned a score of 3 for the item
  23. 23. Group Performance on French Listening Assessment• Group scores are reported for each item by the level of prompt required. Thus, of a total of 18 students who took the test (several students did not complete specific items and received a NA) 3 provided the correct response for item 1 on their initial attempt. They therefore received a score of 4. Three students required one prompt and received a score of 3; two required 2 prompts and received a score of 2; 1 student required three prompts and received a score of 1 and 1 students was unable to provide the correct response after 4 prompts and received a score of 0.
  24. 24. Group Performance on French Listening Assessment• Scores are also reported by rubric. Thus, the first 5 items included in the final row are discourse relevant; the following 9 are relevant to grammar. The 5 items includes under Transfer represent difficult items that are included at the final five items in the test and are designed to determine the extent to which test takers may have learned during the test. Notice that the first two Transfer items (11, 12) are grammatical and the final three (13, 14, 15) are discourse. – Notice that items 11, 12, and 14 were more difficult (as expected) given the low number of students that provided the correct response and their first attempt. Items 13 and 15 were easier as 8 and 6 students respectively provided a correct response on their initial try and only 2 and 4 students respectively required only a single prompt to produce a correct answer.
  25. 25. French Reading: Low Intermediate Group N Mean SD Min Max FRA 23 47.478 11.712 20 60 FRM 23 54.304 6.123 36 60t=4.487 p< .001 ES = 2.151 r = .732
  26. 26. French Reading: High Intermediate Group N Mean SD Min Max FRA 42 40.762 12.81 8 60 FRM 42 51.31 6.809 31 60t = 9.826 p< .001 ES (d = 1.028; r = .457)
  27. 27. French Listening: High Intermediate Group N Mean SD Min Max FLA 34 29.264 15.596 4 59 FLM 34 44.382 11.311 17 59t = 12.529 p< .001 ES (d = 1.109; r = .485)
  28. 28. Chinese Reading: High Intermediate Group N Mean SD Min Max CRA 81 47.654 23.3 8 96 CRM 81 70.802 18.02 10 96t = 19.585 p< .001 ES (d = .1.111; r = .480)
  29. 29. Chinese Listening: Low Intermediate Group N Mean SD Min Max CLA 71 42.592 25.977 0 92 CLM 71 63.535 18.904 7 92t = 15.367 P < .001 ES (d = .922; r = .418)
  30. 30. Learning Potential Scores• “…the learning potential score (LPS) has to reflect both gain made by the student from pre- to post-test and an absolute achievement score at the post-test” (Kozulin& Garb 2002: 121)• LPS Sub-Groups – High > 1.0 – Mid < .99; > .72 – Low < .71
  31. 31. Learning Potential Scores: Chinese Reading Test LPS1.40001.20001.00000.80000.60000.40000.20000.0000 0 10 20 30 40 50 60 70 80 90
  32. 32. LPS for French Listening Chinese Reading AssessmentACTUAL SCORE MEDIATED SCORE LPS 20 45 .729 20 51 .854 20 59 1.020 24 49 .770 24 60 1.000 32 67 1.062 48 75 1.062
  33. 33. Interpretation of LPS• In the previous slide three students had the same ACTUAL score on the Chinese Reading Assessment: 20 – Notice that they each generated a different LPS • One what low -- .729; one was moderate -- .854; and one was high – 1.020 • This indicates that they each reacted differently to mediation provided by the prompt – The first student had a mediated score of 45; the second had a mediated score of 51 and third had a mediated score of 59 – LPS score then predicts how each student is likely to benefit from instruction, with a low LPS pointing to the need for more instruction and a high LPS indicating the need for less instruction in order for the learner to develop. – NB: These predictions must be supported by empirical research.• Notice that two students with different actual scores (i.e., 32 & 48) have identical LPSs even though their respective mediated scores also varied. LPS is determined by relative difference between ACTUAL and MEDIATED score.
  34. 34. Tests Available Soon thru CALPER• General information about dynamic assessment, including working paper and bibliography on CALPER’s webpage PROJECTS• Second Edition of Dynamic Assessment: Teacher’s Guide (available for purchase on DVD for $20)• http://calper.la.psu.edu/dyna_assess.php

×