Retention Study: Efficacy of Online Boating Safety Courses Conference Presentation Richard Deatz Bob Gossman Presented to: National Association of State Boating Law Administrators Wednesday, 30 September 2009

Agenda Project Purpose Retention Study Objectives Retention Study Summary Knowledge Test and Questionnaires Sampling Preliminary Findings Summary and Conclusions

Project Purpose

Retention Study Objectives

Retention Study Summary

Knowledge Test and Questionnaires

Sampling

Preliminary Findings

Summary and Conclusions

Project Purpose To improve classroom and online education specific to recreational boating and inform NASBLA policy development. Creation of Delivery and Presentation Standards Assessment of the Efficacy of Distance Learning Courses (i.e., Retention Study)

To improve classroom and online education specific to recreational boating and inform NASBLA policy development.

Creation of Delivery and Presentation Standards

Assessment of the Efficacy of Distance Learning Courses (i.e., Retention Study)

Retention Study Objectives Initial Plan: To investigate differences between classroom and online courses with regard to initial knowledge acquisition and knowledge retention after 4 months Revised Plan: To investigate participant retention rates for the classroom training condition only, providing baseline data for future study of online training

Initial Plan: To investigate differences between classroom and online courses with regard to initial knowledge acquisition and knowledge retention after 4 months

Revised Plan: To investigate participant retention rates for the classroom training condition only, providing baseline data for future study of online training

Retention Study Summary Volunteers who successfully completed the end of course exam were recruited to participate in study Participants completed a questionnaire and 25 item knowledge test immediately after completing the course Participants were contacted approximately 4 months later to complete a questionnaire and the same 25 item knowledge test on-line Data analyses of difference scores and frequencies were completed

Volunteers who successfully completed the end of course exam were recruited to participate in study

Participants completed a questionnaire and 25 item knowledge test immediately after completing the course

Participants were contacted approximately 4 months later to complete a questionnaire and the same 25 item knowledge test on-line

Data analyses of difference scores and frequencies were completed

Knowledge Test and Questionnaires NASBLA selected 25 test items from the end of training examination for use in data analysis (Reduced Exam) State specific items excluded Items are representative of safety standards The Reduced Exam was administered as part of an End of Course Questionnaire beginning in early March The Retention Questionnaire includes boating experience questions (e.g., time spent boating, accidents or violations) The Reduced Exam was administered again as part of the Retention Questionnaire; administered on-line 4 months later

NASBLA selected 25 test items from the end of training examination for use in data analysis (Reduced Exam)

State specific items excluded

Items are representative of safety standards

The Reduced Exam was administered as part of an End of Course Questionnaire beginning in early March

The Retention Questionnaire includes boating experience questions (e.g., time spent boating, accidents or violations)

The Reduced Exam was administered again as part of the Retention Questionnaire; administered on-line 4 months later

Sampling States from each geographic region (west, north, south) were identified that satisfied the following criteria: Boating Safety training is provided in classroom Classroom providers agree to participate Solicit volunteers Provide volunteers with the questionnaire and examination form Forward completed forms to HumRRO Participating states are: California, Georgia, Oklahoma, Oregon, Pennsylvania, Tennessee, and Virginia Data collection continues through October

States from each geographic region (west, north, south) were identified that satisfied the following criteria:

Boating Safety training is provided in classroom

Classroom providers agree to participate

Solicit volunteers

Provide volunteers with the questionnaire and examination form

Forward completed forms to HumRRO

Participating states are: California, Georgia, Oklahoma, Oregon, Pennsylvania, Tennessee, and Virginia

Data collection continues through October

Findings Current findings are extremely preliminary and are based on: Classroom data collected Feb – April, 2009 Retention data collected through July – Sept, 2009 Data collection and analysis continue and have been extended to: Classroom data collected through October 2009 Retention data collected through March 2010

Current findings are extremely preliminary and are based on:

Classroom data collected Feb – April, 2009

Retention data collected through July – Sept, 2009

Data collection and analysis continue and have been extended to:

Classroom data collected through October 2009

Retention data collected through March 2010

Preliminary Findings

Participant Descriptive Data

Participant Descriptive Data (cont.)

Variable N Mean StdDev TotalScore_A 78 20.974 2.532 TotalScore_B 78 18.935 2.727 Difference_score 78 2.038 3.192 Retention Data – Paired Comparisons t-test of difference scores T-TEST Diff Score (Test A - Test B) t value df p 5.64 77 <.0001

Variable N Mean StdDev

TotalScore_A 78 20.974 2.532

TotalScore_B 78 18.935 2.727

Difference_score 78 2.038 3.192

Retention Level by Age Group

Retention Level by Boating Frequency

Retention Level by Gender

Retention Level by State

Summary and Conclusions Even though there is a statistically significant decrease in test scores between first and second administration of the Reduced Exam, the difference is small (~2 items), and in general, we can say that the average retention rate is pretty good The study does show some interesting findings which, if they still exist with a larger sample, may point the way to additional study Data collection continues

Even though there is a statistically significant decrease in test scores between first and second administration of the Reduced Exam, the difference is small (~2 items), and in general, we can say that the average retention rate is pretty good

The study does show some interesting findings which, if they still exist with a larger sample, may point the way to additional study

Data collection continues

Questions Retention Study: Efficacy of Online Boating Safety Courses Wednesday, 30 September 2009 Richard Deatz, email: [email_address] Bob Gossman, email: bgossman@humrro.org