Test Assessment and Implementation Rationale

609 views
530 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
609
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Test admin informed by test design
  • When reliability issues identified or every 6 months
  • Test Assessment and Implementation Rationale

    1. 1. RELTA in Practice
    2. 2. Overview <ul><li>Test administration </li></ul><ul><li>Scoring processes </li></ul><ul><li>- section scores and weighting </li></ul><ul><li>- rating processes </li></ul><ul><li>Generation of results </li></ul><ul><li>Test maintenance: reliability </li></ul><ul><li>Compromises: theory vs practice </li></ul>
    3. 3. Recap: Structure Listening Speaking
    4. 4. Recap: Delivery
    5. 5. Administration requirements <ul><li>On-site delivery </li></ul><ul><li>Separation of delivery and rating processes </li></ul><ul><li>Quality control: score reliability, training, data collation and storage </li></ul><ul><li>Minimise workload and human error </li></ul><ul><li>Maximise test security </li></ul>
    6. 6. Administration: Roles <ul><li>Administrators - test centre </li></ul><ul><li>Examiners = interlocutors - test centre </li></ul><ul><li>Raters x2 (or 3) – anywhere (local or RMIT) </li></ul><ul><li>RMIT Operations </li></ul>
    7. 7. Administration RELTA Examining RELTA Rating Issuing of results Test centre roles RMIT roles Local administration Local examining RMIT delivery RELTA Examining RELTA Rating Issuing of results RELTA Examining RELTA Rating Issuing of results
    8. 8. Administration Test versions dispatched Results issued: Test Centre Regulator Candidates RELTA administered Speech files rated (x2 raters) Speech files uploaded to server Rater 1 and Rater 2 scores combined and checked Speech file dispatched for independent rating
    9. 9. Administration: Processes <ul><li>Register candidates </li></ul><ul><li>Administer RELTA Listening – in groups </li></ul><ul><li>Listening tests marked – database </li></ul><ul><li>Examiner delivers RELTA Speaking </li></ul><ul><li>Speaking performance recorded </li></ul><ul><li>Uploaded to RMIT server </li></ul><ul><li>Raters (1 and 2) login to database and enter scores </li></ul><ul><li>Rater scores compared for reliability – 3 rd rater </li></ul><ul><li>Results calculated </li></ul><ul><li>Certificates generated </li></ul>
    10. 10. Administration: SRMS <ul><li>Scoring Record Management System (SRMS): </li></ul><ul><li>Developed to manage administration: </li></ul><ul><ul><ul><li>Distributes candidate speaking files to raters </li></ul></ul></ul><ul><ul><ul><li>Captures and rater scores </li></ul></ul></ul><ul><ul><ul><li>Allows for remote and independent rating </li></ul></ul></ul><ul><ul><ul><li>Identifies rater discrepancies </li></ul></ul></ul><ul><ul><ul><li>Generates results </li></ul></ul></ul><ul><ul><ul><li>Produces certificates </li></ul></ul></ul><ul><ul><ul><li>Stores candidate details and speaking files centrally </li></ul></ul></ul><ul><li>Online access for: </li></ul><ul><ul><ul><li>Administrators (registering of candidates, accessing results) </li></ul></ul></ul><ul><ul><ul><li>Raters (enter scores for RELTA speaking) </li></ul></ul></ul>
    11. 11. Scoring 45% 10 Section 3 35% 10 Section 2 20% 10 Section 1 Speaking Weighting Listening (comprehension) Marks
    12. 12. Scoring: Speaking 4 ICAO Score (lowest overall score reported) OVERALL SCORES Weighted average score Section 3 45% Section 2 35% 5 N/A 5 5 4 4 Section 1 20% Interactions Comprehension Fluency Vocabulary Structure Pronunciation
    13. 13. 4 ICAO Score (lowest overall score reported) OVERALL SCORES Weighted average score Section 3 45% 4 N/A 4 4 4 4 Section 2 35% 5 N/A 5 5 4 4 Section 1 20% Interactions Comprehension Fluency Vocabulary Structure Pronunciation
    14. 14. 4 ICAO Score (lowest overall score reported) OVERALL SCORES Weighted average score 5 N/A 5 4 4 4 Section 3 45% 4 N/A 4 4 4 4 Section 2 35% 5 N/A 5 5 4 4 Section 1 20% Interactions Comprehension Fluency Vocabulary Structure Pronunciation
    15. 15. 4 ICAO Score (lowest overall score reported) OVERALL SCORES 4.65 22/30 4.65 4.2 4 4 Weighted average score 5 N/A 5 4 4 4 Section 3 45% 4 N/A 4 4 4 4 Section 2 35% 5 N/A 5 5 4 4 Section 1 20% Interactions Comprehension Fluency Vocabulary Structure Pronunciation
    16. 16. OVERALL Score (Score reported for ICAO purposes) Combined overall scores 4.2 4.65 22/30 4.65 4.55 4.2 4 4 4.2 4 4.2 Weighted average score 4 5 N/A 5 4 4 4 4 4 4 4 Section 3 45% 4 4 N/A 4 5 4 4 4 4 4 4 Section 2 35% 5 5 N/A 5 5 5 4 4 5 4 5 Section 1 20% Interactions Comprehension Fluency Vocabulary Structure Pronunciation
    17. 17. OVERALL Score (Score reported for ICAO purposes) Combined overall scores 4.2 4.65 4.425 22/30 4.65 4.55 4.6 4.2 4 4.1 4 4.2 4.1 4 4.2 4.1 Weighted average score 4 5 N/A 5 4 4 4 4 4 4 4 Section 3 45% 4 4 N/A 4 5 4 4 4 4 4 4 Section 2 35% 5 5 N/A 5 5 5 4 4 5 4 5 Section 1 20% Interactions Comprehension Fluency Vocabulary Structure Pronunciation
    18. 18. 4 OVERALL Score (Score reported for ICAO purposes) 4 5 4 4 4 4 Combined overall scores 4.2 4.65 4.425 22/30 4.65 4.55 4.6 4.2 4 4.1 4 4.2 4.1 4 4.2 4.1 Weighted average score 4 5 N/A 5 4 4 4 4 4 4 4 Section 3 45% 4 4 N/A 4 5 4 4 4 4 4 4 Section 2 35% 5 5 N/A 5 5 5 4 4 5 4 5 Section 1 20% Interactions Comprehension Fluency Vocabulary Structure Pronunciation
    19. 19. Scoring: Rater 1 interface Enter scores for Sections 1, 2 and 3 Download rating scripts Listen to candidate sound file
    20. 20. Scoring: Rater 2 interface Enter scores for Sections 1, 2 and 3 Download rating scripts Listen to candidate sound file
    21. 21. Admin interface Calculates overall scores Determines ICAO Level Imports scores from R1 and R2 Imports Listening score Identifies rater discrepancies
    22. 22. Data collation _______ _______
    23. 23. Rating discrepancies
    24. 24. SRMS: Rating discrepancy
    25. 25. Inter-rater reliability
    26. 26. Inter-rater reliability
    27. 27. Rater re-accreditation
    28. 28. Rater accreditation
    29. 29. Examiner accreditation
    30. 30. Manuals
    31. 31. Summary Testing for ICAO compliance requires: Test instruments that are well designed and can be implemented effectively

    ×