Downloaded 14 times
![Finding: Attitudinal Summary
Positive
Negative
Neutral
Attitude in Chemistry
Positive
Negative
Neutral
Attitude in English Composition
“I am starting to understand why I am
studying on a Friday evening for the first
time in my entire life. :)”
Positive Attitude
“Go for it (un-enrole) [sic]- [two names
removed]. You both know too much already
and you obviously have nothing to gain from
this course. You’ll be doing us “stupid”
students a favor.”
Negative Attitude](https://image.slidesharecdn.com/bigteambigqualdatalittletime0203final-160209233655/75/Big-Qualitative-Data-Big-Team-Little-Time-A-Path-to-Publication-15-2048.jpg)
Uploaded byQSR International
Big Qualitative Data, Big Team, Little Time - A Path to Publication
The document outlines a project by Dr. Charlotte Clark and Noelle Wyman Roth focusing on managing large qualitative data analysis using NVivo, with funding from the Bill and Melinda Gates Foundation. It details the project's aim to evaluate peer-to-peer interactions in writing within massive open online courses (MOOCs), the challenges faced by the team, and the findings related to student learning gains and attitudes in chemistry and English composition forums. The project emphasizes the importance of meticulous planning, team communication, and the iterative nature of qualitative data analysis.
More Related Content
Similar to Big Qualitative Data, Big Team, Little Time - A Path to Publication
Big Qualitative Data, Big Team, Little Time - A Path to Publication
- 1. Big Qualitative Data,Big Team, Little Time – A Path to PublicationFebruary 3, 2016 Dr. Charlotte Clark, Duke University Noelle Wyman Roth, QSR International
- 2. Or, how tomanage a large team project with a lot of data in a short period of time. Overview & Project Data Collection & Management Coding Workshop Findings Lessons Learned
- 3. Overview & Timeline January14 Retained; Funding from Bill & Melinda Gates Foundation February – Early March March 10-14 March 15-16 Planning Training & Intensive Coding Workshop Submission March 17-31 Query & Analysis March 31 Writing & Editing
- 4. Peer-to-Peer Writing inIntroductory Level MOOCs *Source: Wikipedia • Online course aimed at unlimited participation and open access via the web* English Composition I: Achieving Expertise Introduction to Chemistry Massive Open Online Course (MOOC) • evaluate how peer-to-peer interactions through writing impact student learning in introductory-level massive open online courses (MOOCs) across disciplines. • compare humanities and natural sciences class. Project Aim • peer-to-peer interactions in writing through the forums and through peer assessment enhance learner understanding, link to course learning objectives, and generally contribute positively to the learning environment. Results
- 5. Data Collection &Sampling Two sources of data from both courses: 1. Discussion Forum • Weekly forum posts in each course • Top posters in each course • General discussion forum 2. Peer Assessments Data type Hours Basis estimate PIT Threads 60 Basis: 35 threads per PIT x 7 PIT = 240 total threads. Estimate 15 minutes on average per thread to code. Top 3 posters in threads 24 Basis: 4 hours per student, 3 top posters in 2 disciplines. Code no more than 50 posts per student; not looking at threads they initiated, nor at threads to which they posted for context. Only coding their own words. Peer Review Chemistry 8 Basis: 50 students, 10 minutes per student Peer Review Eng IAAW 30 Basis: 2% of 8800 threads; 10 minutes to code each thread Peer Review Eng Procrastinators 15 Basis: 125 threads; 7 minutes to code each thread Peer Review Eng Student portfolios 25 Basis: 50 students (match to Chemistry), 30 minutes per student (2 minutes per eval of project draft and final x 8 + 15 minutes for student reflection)
- 6. Data Collection &Sampling Chemistry English Composition Total Sources Posts Sources Posts Sources Posts Top Posters 3 85 3 209 6 294 Forums 124 1344 206 1051 330 2385 General Forums 25 809 37 86 63 895 Points in Time 99 535 133 768 232 1303 Week 1 29 163 36 106 65 269 Week 4 35 164 35 289 70 453 Week 7 35 208 27 169 62 377 Week 12 N/A N/A 35 204 35 204 Peer Review Writing assignment on forums 106 370 96 325 195 695 Student Portfolios N/A N/A 40 N/A 40 N/A Peer Evaluations N/A N/A 279 N/A 279 N/A Self-Evaluations N/A N/A 39 N/A 39 N/A Total: 592 sources; 3374 posts
- 7. Challenges • Team of9 coders, including PIs • Little to no experience with qualitative data analysis • New to NVivo • One week timeline for data collection, coding, & analysis • Tons of data • Multiple data types • No data gathered • Developed a plan • Coders gathered data based on assignments • Trained only on what need to know • Created a schedule • Tracked progress carefully • Used multiple tools to manage data & collaboration Solutions
- 8. Why NVivo WorkedWell • Go through an intercoder reliability process. • Talk to your teammates frequently. • Keep good notes and track decisions. • Develop a team protocol. • Maintain individual research journals. • Keep a codebook. • Be iterative and flexible. Merge projects Coding comparison query Coding stripes by user Memos See also links Node description box Modify list view In NVivo:In General:
- 9. Create NVivo File Merge projects; compare coding Distributeto coding team Master File Coder 1 Master File Coder 2 Continue Analysis Coder 1 Coder 2 Collaboration in NVivo
- 10. Coding Workshop Timeline •Developed coding protocol • Created master NVivo file Prior to Workshop • Morning: introduced project, manual coding, collaborative coding • Afternoon: ~3 hour NVivo training Day 1 • Morning: team; import & code data • Afternoon: Intercoder reliability discussion; independent coding Day 2
- 11. Coding Workshop Timeline •Individual coding assignments • Daily team discussion/check-in • Set goals & tracked progress Days 3-4 • Analysis discussion • Cleaning data for coding inconsistencies Day 5 • Cross-tabulate & query based on research questionsDays 6-7
- 12. Finding: Student DiscussionsReflect Course Content Course Week 1 Week 4 Week 7 Week 12 Chemistry English Composition
- 13. Finding: General &Discrete Learning Gains in Forums Learning Gains Demonstrates what has learned Evidence of incorporating feedback Improved Grades Understanding Learning Gains in Chemistry Forums Learning Gains Demonstrates what has learned Evidence of incorporating feedback Improved Grades Understanding Learning Gains in English Composition Forums “I was stuck with the idea that my introductions should be one paragraph long. Maybe I should experiment with longer introductions.” “And I feel comfortable enough with the chemistry, the basic chemistry, to not avert my eyes like I used to. Whenever I saw a chemical equation I just, oh well, never mind, and I’d just skip it.” “I don’t know about you, but I’ve already learned an amazing amount from this class!”
- 14. Finding: Learning Gainsin Peer Review Demonstrates what has learned Evidence of incorporating feedback Understanding Learning Gains in Chemistry Peer Review Learning Gains Demonstrates what has learned Evidence of incorporating feedback Learned through providing feedback Understanding Learning Gains in English Composition Peer Review “I found peer comments and their assessment invaluable.” “I am, however, grateful for the kind parts of your review, and willingly admit to faults within the essay, although until this week, I was, like my fellows, unaware of the expected work on electron transits. By the time I did become aware of this, it was too late to make alterations! Thank you for a thoughtful review.” “Even more important bit I learned was the importance of feedback. Feedback provides an opportunity to rethink the project, and dramatically improve it.”
- 15. Finding: Attitudinal Summary Positive Negative Neutral Attitudein Chemistry Positive Negative Neutral Attitude in English Composition “I am starting to understand why I am studying on a Friday evening for the first time in my entire life. :)” Positive Attitude “Go for it (un-enrole) [sic]- [two names removed]. You both know too much already and you obviously have nothing to gain from this course. You’ll be doing us “stupid” students a favor.” Negative Attitude
- 16. Lessons Learned &Limitations Identify what is need to know Keep the lines of communication open Rigorous & deliberate planning is key Limitations to NVivo merging process
- 17.
Editor's Notes
- #3 NEWR
- #4 NEWR February – approached Early March – planning and developed master file Spring break – coding Weekend after – analysis/query March 18-31 – writing & editing March 31- submission Revise and resubmit: 30 July 2014 Resubmitted: 8/22/2014 Final acceptance: 9/24/2014 (say these)
- #5 This study aimed to evaluate how peer-to-peer interactions through writing impact student learning in introductory-level massive open online courses (MOOCs) across disciplines. Results indicate that peer-to-peer interactions in writing through the forums and through peer assessment enhance learner understanding, link to course learning objectives, and generally contribute positively to the learning environment.
- #6 CRC
- #7 Pick one row and walk through as an example: sampled 25 threads, over 800 individual posts What does top poster mean? CRC
- #8 CRC
- #9 Want to add something about how the team interacted with us? Coding conference time? Say: did some limited coding but most of the time we were rotating around the room, available for questions Facilitate collaboration and data management among coders Systematically gather web-based data using NCapture Identify differences in coding/intercoder reliability Code and clean Synthesize multiple data types & compare across courses NEWR
- #10 NEWR
- #11 NEWR Day 1: focus on what people needed to know; trained only on specific parts of NVivo: NCapture, create nodes, code, research journal, see also links; text search query; classifications Day 2 was a “slow coding” day with inter-rater reliability checkins. Wednesday we ramped it up! Thoughts: ● Noelle create a list of keyboard shortcuts that people may like to use? ● Dropbox ID sign-in list ● I created a tiny url for our mooc activity spreadsheet: tinyurl.com/moocdook Monday Morning: 9:00: Settling in 9:15: Introductions 9:45: Project introduction (Denise and Dorian). What is a MOOC, describe the Chemistry and Eng Comp, describe the Research project. Charlotte project the research questions. Take time to really think about the meaning of the research questions with the PI’s. 10:15: Introduction to manual qualitative coding and to our data. a) Define descriptive coding, topical coding, analytical coding. Use brief paragraph b) Moving from individual to collaborative coding. Use two forum threads. i) individual manual coding with RQs in mind. Give them 10 minutes and then ask them to create an individual list of themes, organized how they wish. Go to break when completed that. ??? to 11:00 BREAK 11:00: Continuing manual coding exercise. Put in teams of 3 or 4. Using the personal node lists, create a collaborative node list. Discuss each of the three collaborative lists in turn, comparing similarities and differences. Ask each team if any nodes particularly needed team discussion and why, and which nodes would be more difficult to code reliably in a team. 12:00 LUNCH Monday Afternoon: Working in NVivo 1:00: Introducing NVivo as a tool generally. Open program. Drive around. Prompt for user on launch. 1:15: Import pdf’s we already coded manually. Create research journal. Show see-also links 1:45: Teach NCapture. Show how to print to pdf. Note that should use Explorer, because that lets you determine where you put your ncaptured files. 2:15: Beginning coding 2:30: BREAK 2:45: Nodes and coding. Source coding to multiple nodes at once. 3:15: Text search queries 3:30: Introduce classifications 3:45: Demonstrate coder comparison query using Down East Sample project Tuesday: Beginning our work The morning will go at a deliberate pace as we work to ensure understanding and reliability. 9:00: Remind them of our research questions. Point out research question poster. 9:15: Distribute NVivo files (have each go to dropbox and download their file onto their desktop). Each creates a folder on their desktop for MOOC files. Open NVivo files. Go to memo’s and have each person create a research journal with their first name in the title. Show them source folder structure. Spend time going over node structure and protocols. 9:45: Coding to node structure: Collaboratively code one thread that is already in their source file. Show how they can refer to the protocol. Demo how to classify the sources. Demo how to populate the classification sheet using web data. Demo how to code sources to Forum nodes. Demo how to code entire posts to the post nodes. Give them time to code partial or complete posts to the content nodes. Have them turn on user stripes and each team of 3-4 talks about the experience. 10:15: Collecting data and importing: In same teams from Monday of three or four, have each team print to pdf a set of two forum threads (different ones for each team? LIkely just go to one subforum (maybe Chemistry week 6) and give the first two to one team, second two to the second, etc). Specifically, go through with each the process of finding the first designated thread, printing as pdf, making note of where it is saved. Go through importing into NVivo. Have them individually do a second designated thread without prompting. 10:30: BREAK 10:45: Everyone individually codes the two threads they have imported using protocol we demo’d before break. 11:45: Bring them back together. ● Remind them of this 5 step process for Forum coding: 1. Capture: Capture thread as pdf using “print to pdf” function. We suggest capturing multiple at one time -- perhaps 10? 2. Import: Import into NVivo. 3. Classify: Classify as thread (can do multiple sources at one time). Populate classification sheet using data on forum home page (where threads are listed) 4. Code whole sources/threads to thread organization nodes 5. Code whole posts to post structure nodes 6. Code whole or partial posts to content nodes 7. Throughout, write in research journal. Make note of questions for the team. Look for pithy quotes. ● Have them upload their files to dropbox for our use during lunch. 12:00-2:00 LUNCH. We combine files, look at results. determine what are problems and what to discuss. 2:00: Show them results. Talk about strengths, challenges. Go over the protocol by which they each individually now will know which fora to code. Set them off! 4:30: Bring them back together. What went well and what went poorly. How much did each coder get done.
- #12 NEWR
- #13 NEWR
- #14 Give an example: one of our nodes was “demonstrates what has learned,” which we defined as ____ and example of what was coded to these nodes. “Demonstrates what is learned” Here are a couple examples. First, I thought that I am not a writer because a writer in my mind who is write something that published but after watched the video I just realized that I am a writer also even not publish my writing. I have learnt how important is the introduction of the case study when introduction provides the hook to the reader, the reader is more interested in reading the entire article. I learn how to find some jokes and interesting quotes at the very beginning tio evoke the readers interest." “Understanding” “…it balances easily now. I’ve learned my lesson.” “aaah Octet Rule! Thanks [names removed] “ah right… always need to keep in mind that goal of the lowest overall energy. Good good…penny beginning to drop.” CRC
- #15 “Evidence of Incorporating feedback” Note that we accepted past tense and future intention “I tried it the way you described” “I’ll research it further as you suggested” “Yes I will consider…. And I will calculate the ….” “I;m going to take into account your points” CRC
- #16 Every post or part of a post was coded as either positive, negative, or neutral Majority was neutral, relatively small % was coded as positive “The attitude expressed in student posts was generally more positive than negative in both courses: 2.8 times more positive than negative in Chemistry and 3.9 times more positive than negative in English Composition.” Positive attitude: “I am starting to understand why I am studying on a Friday evening for the first time in my entire life. :)” Negative attitude: “Go for it (un-enrole) [sic]- [two names removed]. You both know too much already and you obviously have nothing to gain from this course. You’ll be doing us “stupid” students a favor.” CRC
- #17 CRC (NEWR to chime in)