Presented by Tim Brandwood and Damien Gouriet.
Codeit have been involved in Verbatim Coding and Machine Learning for the last 7 years.
In this presentation, they share their research on the use of Generative AI for coding market research verbatim responses, and their experiments using Generative AI to automate the coding process.
They also discuss the strengths and limitations of this as a coding tool, the role of human oversight in the process, and the potential for customized machine learning models working alongside it.
The highlights the importance of human expertise and the use of GPT as a labour-saving tool rather than a complete replacement for humans in the market research process.
Access the presentation recording via NewMR.org/Play-Again.
3. THE PURPOSE OF CODING
Open-ended Comments
I like the orange taste,
but it’s too sweet
The sweetness is a bit much
I think the orange colour is
too pale and artificial looking
I don’t like the bits of
orange in it
I don’t like the idea of
artificial sweeteners
Analysable Quantitative Measures
Taste (871) 87%
Too Sweet (632) 63%
Tastes Artificial (421) 42%
Doesn’t taste like real orange (237) 24%
Appearance (423) 42%
Colour – too light/pale (302) 30%
Colour – too artificial looking (150) 15%
Colour – too dark (100) 10%
Colour – other comments (40) 4%
Ingredients (408) 41%
Artificial Sweeteners (300) 30%
Real Orange (150) 15%
Orange bits (50) 5%
Packaging (387) 39%
Bottle too big (223) 22%
Difficult to hold (150) 15%
Label hard to read (87) 8%
Dislike bottle shape (60) 6%
Requires: Accuracy and Precision
4. Extracting Themes
Complete Coding:
Generates rich, human-like themes
Performs extremely well:
CAN CHATGPT DO CODING THEN?
Quantifying Themes
“Chat” interactivity allows you to ask follow-up questions,
and dig deeper
Complete Coding:
Incomplete coverage
Struggles to accurately and precisely quantify themes. Issues include:
Miscoding
Repeatability
20% verbatims coded
80% accuracy
5. PEOPLE STILL NEED TO BE
INVOLVED IN THE PROCESS
Go further, quicker …but you still need to pedal!
ChatGPT is an electric bike for your mind.
-Richard Bowman (https://prompt.mba)
12. ChatGPT is a big step forward – but only truly useful for coding up to a point
Human oversight and quality tools needed
Custom ML models outperform ChatGPT
CONCLUSION
Together these developments point towards better, more enriched research
13. Q & A
Ray Poynter
NewMR
Tim Brandwood
Digital Taxonomy
Damien Gouriet
Digital Taxonomy