A Standardized Approach to Assessment – Phase 2<br />New Improvements Based on Student , Consultant & Library Faculty Feed...
In 2008 / 2009, We Began By:<br />Developing and testing an assessment tool that could:<br />Facilitate the development of...
In 2010, We Asked the Question:How can we improve the tool and the assessment process?<br />Usability testing sessions wit...
1. Usability Testing<br />
Usability Sessions: Changes you asked for <br />Comment: I didn’t understand where I was when I entered the tool.<br />
Usability Sessions: Changes you asked for <br />Remedy: A clearly defined menu as the starting screen<br />
Usability Sessions: Changes you asked for <br />Comment: Make it easier to toggle from my personal account to all question...
Usability Sessions: Changes you asked for <br />Remedy:<br /><ul><li> “View Other Librarians’ Questions” was changed to “V...
It’s easier to add and remove questions.
The “My Questions” view and “All Questions” view are different colors for more distinction.
We have indicated which questions have been standardized.</li></li></ul><li>Usability Sessions: Changes you asked for <br ...
Usability Sessions: Changes you asked for <br />Remedy: A message indicates the survey was created and the URL to the surv...
2. More Cognitive Interviews<br />
More Cognitive Interviews:<br /><ul><li>Cognitive interviewing is a survey development technique that involves one-on-one ...
During the cognitive interview process, the interviewer uses techniques to elicit feedback on survey questions.
We asked interviewees to read questions aloud and tell us what they thought the questions meant. Also, whether there was a...
Wanted to aim for 10 interviews prior to Fall 2010 when we planned to use the tool again in the classroom</li></li></ul><l...
We changed “Education students” to “students in the College of Education” to be clearer in our meaning.</li></li></ul><li>...
Cognitive Interviews:Changes we made<br />Question:<br /><ul><li>In PsycInfo, how do I save a search result for the curren...
How can you tell you are reading a magazine instead of a scholarly source of information?</li></li></ul><li>3. Improved St...
Improved Statistical Analysis:Built in TTEST Reports<br /><ul><li>We built in TTEST Reports through a link to an excel spr...
Asteriks by values in the matrix show statistically significant relationships between background factors.
This data set shows that students who received prior library instruction had higher pre-scores than those who had not. Als...
Focus Group:Points for Discussion<br /><ul><li>Are we testing teaching effectiveness or student learning?
Do numbers offer the big picture of teaching effectiveness?
Upcoming SlideShare
Loading in …5
×

Phase II Overview of Information Literacy Assessment Project

436 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
436
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Phase II Overview of Information Literacy Assessment Project

  1. 1. A Standardized Approach to Assessment – Phase 2<br />New Improvements Based on Student , Consultant & Library Faculty Feedback<br />Shannon M. Staley<br />Education & Web Services Librarian<br />
  2. 2. In 2008 / 2009, We Began By:<br />Developing and testing an assessment tool that could:<br />Facilitate the development of pre and post achievement tests<br />Link survey questions to learning outcomes and ACRL standards<br />Deliver real time reports and statistical analysis<br />
  3. 3. In 2010, We Asked the Question:How can we improve the tool and the assessment process?<br />Usability testing sessions with librarians who have not used the tool<br />More cognitive interviews with students to improve question development<br />Improved statistical analysis <br />Focus group with librarians who have used the tool<br />
  4. 4. 1. Usability Testing<br />
  5. 5. Usability Sessions: Changes you asked for <br />Comment: I didn’t understand where I was when I entered the tool.<br />
  6. 6. Usability Sessions: Changes you asked for <br />Remedy: A clearly defined menu as the starting screen<br />
  7. 7. Usability Sessions: Changes you asked for <br />Comment: Make it easier to toggle from my personal account to all questions and possibly select and copy some for my account.<br />
  8. 8. Usability Sessions: Changes you asked for <br />Remedy:<br /><ul><li> “View Other Librarians’ Questions” was changed to “View All Questions.”
  9. 9. It’s easier to add and remove questions.
  10. 10. The “My Questions” view and “All Questions” view are different colors for more distinction.
  11. 11. We have indicated which questions have been standardized.</li></li></ul><li>Usability Sessions: Changes you asked for <br />Comment: I didn’t understand when the survey was actually created – also the link to the survey was on a site somewhere.<br />
  12. 12. Usability Sessions: Changes you asked for <br />Remedy: A message indicates the survey was created and the URL to the survey is provided so that it can be posted anywhere.<br />
  13. 13. 2. More Cognitive Interviews<br />
  14. 14. More Cognitive Interviews:<br /><ul><li>Cognitive interviewing is a survey development technique that involves one-on-one interviews with a sample of the target population to assess the quality of survey questions.
  15. 15. During the cognitive interview process, the interviewer uses techniques to elicit feedback on survey questions.
  16. 16. We asked interviewees to read questions aloud and tell us what they thought the questions meant. Also, whether there was any terminology that was confusing.</li></li></ul><li>More Cognitive Interviews:<br /><ul><li>Original pilot study involved only 5 one-on-one interviews with undergraduate students
  17. 17. Wanted to aim for 10 interviews prior to Fall 2010 when we planned to use the tool again in the classroom</li></li></ul><li>Cognitive Interviews:Changes we made<br />Question:<br /><ul><li>What are a few of the core databases for Education students?</li></ul>  <br />Changes:<br /> <br /><ul><li>We changed “core databases” to “most important databases” after we learned that some interviewees were confused by the original term.
  18. 18. We changed “Education students” to “students in the College of Education” to be clearer in our meaning.</li></li></ul><li>Cognitive Interviews:Changes we made<br />Question:<br /><ul><li>When you start a search, why is it best not to limit results </li></ul> to full text only?<br />  <br />Changes:<br /> <br /><ul><li>We added “articles” after “full text” in the question after</li></ul> we learned that some interviewees weren’t clear on the meaning of the phrase “full text.”<br />
  19. 19. Cognitive Interviews:Changes we made<br />Question:<br /><ul><li>In PsycInfo, how do I save a search result for the current search session?</li></ul>  <br />Changes:<br /> <br /><ul><li>We made explicit that PsycInfo was a database, since many of the interviewees did not know what the term meant.</li></li></ul><li>Cognitive Interviews:Changes we made<br />Question:<br /><ul><li>How can you tell you are reading a popular magazine?</li></ul>  <br />Changes:<br /> <br /><ul><li>Interviewees thought “popular” meant desirable or heavily used material rather than non-scholarly so we changed the question to:
  20. 20. How can you tell you are reading a magazine instead of a scholarly source of information?</li></li></ul><li>3. Improved Statistical Analysis<br />
  21. 21. Improved Statistical Analysis:Built in TTEST Reports<br /><ul><li>We built in TTEST Reports through a link to an excel spreadsheet</li></li></ul><li>Improved Statistical Analysis:Built in TTEST Reports <br /><ul><li>Snapshot of the an excel spreadsheet report with TTEST tabulation</li></li></ul><li>Improved Statistical Analysis:Advanced Correlation Templates<br /><ul><li>Kendall’s Bivariate Correlation Analysis
  22. 22. Asteriks by values in the matrix show statistically significant relationships between background factors.
  23. 23. This data set shows that students who received prior library instruction had higher pre-scores than those who had not. Also those who received prior library instruction spent more time using library resources than those who had not.</li></li></ul><li>4. Focus Group<br />
  24. 24. Focus Group:Points for Discussion<br /><ul><li>Are we testing teaching effectiveness or student learning?
  25. 25. Do numbers offer the big picture of teaching effectiveness?
  26. 26. Need more research to develop a cut score or acceptable percentage of correct responses.
  27. 27. Need to gather support from other SJSU librarians and get campus endorsement
  28. 28. The data is helpful to librarians and faculty – it gives us some ballpark idea of whether students are learning or not</li></li></ul><li>Web Site Tool Kit:<br />URL:<br />http://www.sjsu.edu/faculty/sstaley/sila/<br /><ul><li>Provides complete overview of the entire assessment project from pilot study to present
  29. 29. Complete data collected from usability testing sessions
  30. 30. Complete data collected from focus group</li></li></ul><li>Questions?<br />

×