The document details a content analysis methodology for evaluating open-ended survey responses in a teaching information technology literacy course. It outlines steps for gathering, parsing, categorizing, and achieving inter-rater reliability on active learning techniques used in the course. Additionally, it provides examples from a related study on academic librarians' use of virtual information services, focusing on reported successes and challenges.

1. Content Analysis of
Open-Ended Responses
LIS 665 Teaching Information Technology Literacy * Dr. Diane Nahl
University of Hawaii * LIS Program * Spring 2013

2. Content Analysis Methodology
 Classification Analysis of Patterns and Themes
 Complete document analysis
 Learner-generated discourse, comments, and narratives
 Filtering narrative responses, comments, and expressions data via
naturally occurring clusters and making category titles
 Elemental Analysis of Words and Phrases
 Parsing responses into word, phrase, and concept elements to define
“occurrences”
 Word or concept frequencies
 Percents or proportions of expressions in categories
 Achieving and Defining Agreement
 95% or better needed for inter-rater reliability
 Content Analysis Steps 1-7 + Step 8 (Radcliff et al. Ch 13, pp. 146-151)
Nahl 2013 2 LIS 665 Teaching Information Technology Literacy

3. Content Analysis of Minute Writing Data: Question 2.
What active learning methods have been used in LIS 665?
1. Copy and paste all of the responses to Question 2 in the raw
data document into a blank document, or save a copy of the
file to work on.
2. Read through all of the responses to Question 2 without
marking any text.
3. Parsing: Read through all of the responses to Question 2 a
second time and use a highlight color or bolding or other
marking method to Parse the names of active learning
methods mentioned in each response. Mark each mention of
any technique.
Nahl 2013 3 LIS 665 Teaching Information Technology Literacy

4. Content Analysis of Minute Writing Data: Question 2.
What active learning techniques have been used in LIS 665?
4. Parsing: Read through a third time to complete marking
each mention of methods. You might have to mark longer
phrases if responses describe instead of name methods.
5. Filtering: Read through the marked text and determine if
the methods fall into more specific categories, e.g., models
are more general than techniques so could form two
categories. Elements of techniques could form another
category. It is up to you to decide on the meaningful
clustering and naming of categories.
6. Coding: Use different highlight colors to separate parsed
text into categories.
Nahl 2013 4 LIS 665 Teaching Information Technology Literacy

5. Content Analysis of Minute Writing Data: Question 2.
What active learning techniques have been used in LIS 665?
7. Categories: Copy and paste the parsed terms into category
listings on the document.
8. Inter-rater agreement: Show your categories to one other
person in class. Discuss any differences you detect in your
parsed terms, category groupings, and names for
categories.
9. Achieving inter-rater agreement: Come to agreement on
categories and groupings by standardizing a definition for
what each category stands for and write the definition under
each category name, above the unique parsed elements.
Nahl 2013 5 LIS 665 Teaching Information Technology Literacy

6. Example: Content Analysis for a
Published Research Article
 An online survey was sent to several librarian lists and Second
Life librarian groups. The open-ended survey responses were
analyzed with content analysis and published:
 Ashford, Robin, Beth Kraemer, Denise Cote and Diane Nahl. 2012.
Academic Librarians and Virtual Information Services in Second Life.
Journal of Library Innovation (January) 3 (1): 20-47.
http://www.libraryinnovation.org/article/view/87
 The following slides (7-12) illustrate portions of the processes
involved in analyzing the open-ended comments of librarians
using Second Life in their professional endeavors.
Nahl 2013 6 LIS 665 Teaching Information Technology Literacy

7. Open-Ended Raw Survey Data
Nahl 2013 7 LIS 665 Teaching Information Technology Literacy

8. Analyzing the Results: Parsing and Coding
Phrases, Inter-Rater Reliability
Nahl 2013 8 LIS 665 Teaching Information Technology Literacy

9. Analyzing the Results:
Categorizing the Comments
1. Successes
o Collaboration & Connecting
o Professional Development
o Reference & Instruction Opportunities
o Socializing
o Content Creation
2. Obstacles
o Technical Difficulties
o Steep Learning Curve
o Insufficient Value
o Unknown Application
Nahl 2013 9 LIS 665 Teaching Information Technology Literacy

10. Figure 3. Reported Successes
Nahl 2013 10 LIS 665 Teaching Information Technology Literacy

11. Figure 4. Reported Challenges
Nahl 2013 11 LIS 665 Teaching Information Technology Literacy

12. Table 1: Academic Librarians in Second Life:
Five Major Role Categories and Four Areas of Difficulty
N comments made in Successes Categories % Total %
each category
35 Collaboration & Connecting 29 %
42 Professional Development 31 %
54 Reference & Instruction 36 %
2 Socializing 1%
19 Content Creation 13 %
152 TOTAL 100 % 152 77%
Challenges Categories
6 Technical Difficulties 13 %
12 Steep Learning Curve 26 %
16 Insufficient Value 35 %
12 Unknown Application 26 %
46 TOTAL 100 % 46 23 %
Grand Total All Comments 198 100 %
Ashford, Robin, Beth Kraemer, Denise Cote and Diane Nahl. 2012. Academic Librarians and Virtual Information Services in Second Life.
Journal of Library Innovation (January) 3 (1): 20-47. http://www.libraryinnovation.org/article/view/87

13. Next Week
 Radcliff et al. Ch 14; ACRL 2003, 2007; IL IQ
 Bring your content analysis results to class for an exercise in
summarizing textual/comment data
 Graphing numerical results
 Bring your numerical data to class in a spreadsheet for an exercise
in making and discussing figures and tables
 Instruction Unit due April 4
 Part I: Individual report on all elements
 Part II: All Team member names on cover page.
 Includes all materials used in session, instructional sequence, keys to
worksheet scoring; and all activities, worksheets, and evaluations keyed
to outcomes (ACRL SPIOs and ACS codes).
Nahl 2013 13 LIS 665 Teaching Information Technology Literacy