SlideShare a Scribd company logo
1 of 23
Download to read offline
Using Generative AI to Assess the Quality
of Open-Ended Responses in Surveys
NewMR Webinar June 2023
Karine Pepin
2CV Research
LONDON I LOS ANGELES I SAN FRANCISCO I SINGAPORE
Using Generative
AI to Assess the
Quality of Open-
Ended Responses
in Surveys
By Karine Pepin
SVP
3
Current Challenges
with Open-Ended
Responses
How Generative AI
can help
Best practices
4
Checks of Data Integrity
Source: Insights Association Data Integrity Initiative A Toolkit from the Insights Association In
Partnership with a Council of Industry Leader
TIME
SPENT
Pre-
survey
In-survey QC
checks
(Automated)
Post-survey checks
(mostly manual)
o Geo-Location Tracking
o Digital Device Fingerprinting
o IP-Deduplication
o Bots
o Bot Script
o Click-Farm
o Form Filler Check
o Instructional Manipulation Check
o Low Incidence Check
o Speeder/Racer
o Straightlining
o Red Herring/Trap Question
o Contradictory Answers
o Open-End Response Validation Survey
o Data Discrepancy Checks
o Open-End Response Review
o Benchmark Comparison
o MaxDiff Question Check
5
In-survey validation:
What it does
Type of issues flagged in survey
• Copy and paste
• Profanity
• Proper grammar
• Duplicate response
• Gibberish
Be D add is do ha
sh he be sh he be
can ha DJ ha sh ha
sh ha can be sh vs
CZ sh ha en be xxx
Amazon all very
good and very nice
debit cards in very
nice and very good
information
technology
they proide
beat service in
short best of
time
because I feel like
you are doing well
da*n I have no
problem getting to
the money
5
6
In-survey
validation: What it
doesn’t do
Thinking back to the last
purchase you made using this
type of payment, where was
the purchase made and why
did you choose this specific
form of payment?
In his inaugural
address, Donald J.
Trump maintained
the “Make America
Great Again”
campaign mantra
that helped sweep
him into office. And
he took the theme of
“transfer of power” a
step further, saying
“we are transferring
power from
Washington
I’m just
making
More
money and
I’m Saving
up even
more
No thanks I don’t
want to answer
this question
again I just got a
call from the lord
and I am in your
high school.
Tesla company is
growing very fast
and they make the
best electric vehicles
in the United States.
Very much I
appreciate
you feeling
today I am
Good night
too much for
your help
today to get a
good one to
come over
and watch
7
Open-Ended Response: What we need
OpenAI's natural language processing models are designed to understand the context in which language is used
allowing them to analyze the meaning and intent behind open-ended responses.
8
Here’s a simple example using ChatGPT
9
Limitations with ChatGPT
Not possible to provide additional
training data to ChatGPT directly
Using GPT via Google Apps Script solves these challenges
Cannot be integrated directly into
Google Sheets/ Excel which allows
you allows you to analyze more than
one open-ended at the time making
it hard to work with large datasets.
Doesn’t get smarter: Doesn’t scale
10
Automation Process
Automate
Use Google Apps Script to build
a custom VERIFY_RESPONSE()
formula that calls the OpenAI
Chat completion API to receive
a YES/NO assessment along
with a reason
Train
Provide open-ended responses
as training data to help the
model learn to generate a more
accurate assessment to similar
open-ended responses
Iterate
Run open-ended
responses in
batches in order
improve on your
training
11
Google Sheet Set-Up: Settings
12
Google Sheet Set-Up: Training Data
13
Google Sheet Set-Up: Survey Question
14
Google Sheet Set-Up: Open-Ended Response Input/Output
15
Google Sheet Set-Up: Open-Ended Response Input/Output
16
Google Sheet Set-Up: Open-Ended Response Input/Output
17
Google Sheet Set-Up: Double API Call for Other Languages
18
How Does it Perform
Is it better than
a human? No.
Is it better
than doing
nothing Yes!
Is it faster
than a
human? Yes!
19
“Training
Makes Perfect”
20
Best Practices
Use generic OE
questions designed
for QC only
Data
Confidentiality
Create a bank of OE
questions so the
model doesn’t have
to be train each time
Efficiency
Use more than one
OE questions to
make the final
assessment
Accuracy
Provide additional
training data when
new countries,
languages, or
audiences are added
Tuning
Review rejected
responses carefully.
Spot check accepted
responses.
Validation
The API could be
integrated directly
into the survey
script, but exercise
caution!
Survey
Integration
Thank you
Karine.Pepin@2cv.com
Q & A
Ray Poynter
NewMR
Karine Pepin
2CV Research
Sponsors

More Related Content

What's hot

Chat GPT TEL Community of Practice
Chat GPT TEL Community of PracticeChat GPT TEL Community of Practice
Chat GPT TEL Community of PracticePeter Windle
 
Generative Artificial Intelligence and Academic Integrity - LIR HEAnet User G...
Generative Artificial Intelligence and Academic Integrity - LIR HEAnet User G...Generative Artificial Intelligence and Academic Integrity - LIR HEAnet User G...
Generative Artificial Intelligence and Academic Integrity - LIR HEAnet User G...Thomas Lancaster
 
Unleashing The Power Of CHAT-GPT
Unleashing The Power Of CHAT-GPTUnleashing The Power Of CHAT-GPT
Unleashing The Power Of CHAT-GPTDemetris D-Papa
 
The Future of AI is Generative not Discriminative 5/26/2021
The Future of AI is Generative not Discriminative 5/26/2021The Future of AI is Generative not Discriminative 5/26/2021
The Future of AI is Generative not Discriminative 5/26/2021Steve Omohundro
 
Can We Fake Academic Integrity? Keynote presentation for Canadian Symposium o...
Can We Fake Academic Integrity? Keynote presentation for Canadian Symposium o...Can We Fake Academic Integrity? Keynote presentation for Canadian Symposium o...
Can We Fake Academic Integrity? Keynote presentation for Canadian Symposium o...Thomas Lancaster
 
Innnovations in online teaching and learning: CHatGPT and other artificial as...
Innnovations in online teaching and learning: CHatGPT and other artificial as...Innnovations in online teaching and learning: CHatGPT and other artificial as...
Innnovations in online teaching and learning: CHatGPT and other artificial as...Rebecca Ferguson
 
The AI Revolution - Into the Age of Imagination - Tony Aubé at WAQ 23
The AI Revolution - Into the Age of Imagination - Tony Aubé at WAQ 23The AI Revolution - Into the Age of Imagination - Tony Aubé at WAQ 23
The AI Revolution - Into the Age of Imagination - Tony Aubé at WAQ 23Tony Aubé
 
Generative AI Use-cases for Enterprise - First Session
Generative AI Use-cases for Enterprise - First SessionGenerative AI Use-cases for Enterprise - First Session
Generative AI Use-cases for Enterprise - First SessionGene Leybzon
 
How ChatGPT and AI-assisted coding changes software engineering profoundly
How ChatGPT and AI-assisted coding changes software engineering profoundlyHow ChatGPT and AI-assisted coding changes software engineering profoundly
How ChatGPT and AI-assisted coding changes software engineering profoundlyPekka Abrahamsson / Tampere University
 
ChatGPT_Prompts.pptx
ChatGPT_Prompts.pptxChatGPT_Prompts.pptx
ChatGPT_Prompts.pptxChakrit Phain
 
Building an AI Startup
Building an AI StartupBuilding an AI Startup
Building an AI StartupTathagat Varma
 
How to Use AI (Like ChatGPT & Bard) in your SEO & Content - A Comprehensive S...
How to Use AI (Like ChatGPT & Bard) in your SEO & Content - A Comprehensive S...How to Use AI (Like ChatGPT & Bard) in your SEO & Content - A Comprehensive S...
How to Use AI (Like ChatGPT & Bard) in your SEO & Content - A Comprehensive S...Volume Nine
 
SPEAK with CHATGPT 24h in US Language
SPEAK with CHATGPT 24h in US LanguageSPEAK with CHATGPT 24h in US Language
SPEAK with CHATGPT 24h in US LanguageErol GIRAUDY
 
Hala GPT - Samer Desouky.pdf
Hala GPT - Samer Desouky.pdfHala GPT - Samer Desouky.pdf
Hala GPT - Samer Desouky.pdfSamer Desouky
 
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...Alain Goudey
 
Generative AI For Everyone on AWS.pdf
Generative AI For Everyone on AWS.pdfGenerative AI For Everyone on AWS.pdf
Generative AI For Everyone on AWS.pdfManjunatha Sai
 
What Leaders Need To KNOW & DO About Generative AI.pdf
What Leaders Need To KNOW & DO About Generative AI.pdfWhat Leaders Need To KNOW & DO About Generative AI.pdf
What Leaders Need To KNOW & DO About Generative AI.pdfSwitch On | Thrive Your Future
 

What's hot (20)

Chat GPT TEL Community of Practice
Chat GPT TEL Community of PracticeChat GPT TEL Community of Practice
Chat GPT TEL Community of Practice
 
ChatGPT.pptx
ChatGPT.pptxChatGPT.pptx
ChatGPT.pptx
 
Generative Artificial Intelligence and Academic Integrity - LIR HEAnet User G...
Generative Artificial Intelligence and Academic Integrity - LIR HEAnet User G...Generative Artificial Intelligence and Academic Integrity - LIR HEAnet User G...
Generative Artificial Intelligence and Academic Integrity - LIR HEAnet User G...
 
Unleashing The Power Of CHAT-GPT
Unleashing The Power Of CHAT-GPTUnleashing The Power Of CHAT-GPT
Unleashing The Power Of CHAT-GPT
 
The Future of AI is Generative not Discriminative 5/26/2021
The Future of AI is Generative not Discriminative 5/26/2021The Future of AI is Generative not Discriminative 5/26/2021
The Future of AI is Generative not Discriminative 5/26/2021
 
Can We Fake Academic Integrity? Keynote presentation for Canadian Symposium o...
Can We Fake Academic Integrity? Keynote presentation for Canadian Symposium o...Can We Fake Academic Integrity? Keynote presentation for Canadian Symposium o...
Can We Fake Academic Integrity? Keynote presentation for Canadian Symposium o...
 
Jim Lecinski - Capturing the Power of AI in Marketing.pdf
Jim Lecinski - Capturing the Power of AI in Marketing.pdfJim Lecinski - Capturing the Power of AI in Marketing.pdf
Jim Lecinski - Capturing the Power of AI in Marketing.pdf
 
Innnovations in online teaching and learning: CHatGPT and other artificial as...
Innnovations in online teaching and learning: CHatGPT and other artificial as...Innnovations in online teaching and learning: CHatGPT and other artificial as...
Innnovations in online teaching and learning: CHatGPT and other artificial as...
 
The AI Revolution - Into the Age of Imagination - Tony Aubé at WAQ 23
The AI Revolution - Into the Age of Imagination - Tony Aubé at WAQ 23The AI Revolution - Into the Age of Imagination - Tony Aubé at WAQ 23
The AI Revolution - Into the Age of Imagination - Tony Aubé at WAQ 23
 
Generative AI Use-cases for Enterprise - First Session
Generative AI Use-cases for Enterprise - First SessionGenerative AI Use-cases for Enterprise - First Session
Generative AI Use-cases for Enterprise - First Session
 
How ChatGPT and AI-assisted coding changes software engineering profoundly
How ChatGPT and AI-assisted coding changes software engineering profoundlyHow ChatGPT and AI-assisted coding changes software engineering profoundly
How ChatGPT and AI-assisted coding changes software engineering profoundly
 
ChatGPT_Prompts.pptx
ChatGPT_Prompts.pptxChatGPT_Prompts.pptx
ChatGPT_Prompts.pptx
 
Building an AI Startup
Building an AI StartupBuilding an AI Startup
Building an AI Startup
 
How to Use AI (Like ChatGPT & Bard) in your SEO & Content - A Comprehensive S...
How to Use AI (Like ChatGPT & Bard) in your SEO & Content - A Comprehensive S...How to Use AI (Like ChatGPT & Bard) in your SEO & Content - A Comprehensive S...
How to Use AI (Like ChatGPT & Bard) in your SEO & Content - A Comprehensive S...
 
SPEAK with CHATGPT 24h in US Language
SPEAK with CHATGPT 24h in US LanguageSPEAK with CHATGPT 24h in US Language
SPEAK with CHATGPT 24h in US Language
 
Hala GPT - Samer Desouky.pdf
Hala GPT - Samer Desouky.pdfHala GPT - Samer Desouky.pdf
Hala GPT - Samer Desouky.pdf
 
The Creative Ai storm
The Creative Ai stormThe Creative Ai storm
The Creative Ai storm
 
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
 
Generative AI For Everyone on AWS.pdf
Generative AI For Everyone on AWS.pdfGenerative AI For Everyone on AWS.pdf
Generative AI For Everyone on AWS.pdf
 
What Leaders Need To KNOW & DO About Generative AI.pdf
What Leaders Need To KNOW & DO About Generative AI.pdfWhat Leaders Need To KNOW & DO About Generative AI.pdf
What Leaders Need To KNOW & DO About Generative AI.pdf
 

Similar to Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys

Bba q&a study final white
Bba q&a study final whiteBba q&a study final white
Bba q&a study final whiteGreg Sterling
 
Growth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - AntwerpGrowth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - AntwerpThibault Imbert
 
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
[DSC Europe 22] Avoid mistakes building AI products - Karol PrzystalskiDataScienceConferenc1
 
How to Correctly Use Experimentation in PM by Google PM
How to Correctly Use Experimentation in PM by Google PMHow to Correctly Use Experimentation in PM by Google PM
How to Correctly Use Experimentation in PM by Google PMProduct School
 
DIT Digitial Marketing Forum: Analytics
DIT Digitial Marketing Forum: AnalyticsDIT Digitial Marketing Forum: Analytics
DIT Digitial Marketing Forum: AnalyticsLar Veale
 
Creating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvementCreating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvementBen Dressler
 
A Happy Marriage between Context-Driven and Agile
A Happy Marriage between Context-Driven and AgileA Happy Marriage between Context-Driven and Agile
A Happy Marriage between Context-Driven and AgileIlari Henrik Aegerter
 
JDO 2019: Data Science for Developers - Matthew Renze
JDO 2019: Data Science for Developers -  Matthew RenzeJDO 2019: Data Science for Developers -  Matthew Renze
JDO 2019: Data Science for Developers - Matthew RenzePROIDEA
 
08-Task Analysis.pdf
08-Task Analysis.pdf08-Task Analysis.pdf
08-Task Analysis.pdfshreyakale11
 
Usability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working GroupUsability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working GroupNormanMendoza
 
Intuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation ProgramIntuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation ProgramOptimizely
 
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...Grid Dynamics
 
The Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMThe Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMProduct School
 
Rinse and Repeat : The Spiral of Applied Machine Learning
Rinse and Repeat : The Spiral of Applied Machine LearningRinse and Repeat : The Spiral of Applied Machine Learning
Rinse and Repeat : The Spiral of Applied Machine LearningAnna Chaney
 
Understand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakesUnderstand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakesTheFamily
 
Breathing the breath of the monster combining agile and context-driven
Breathing the breath of the monster   combining agile and context-drivenBreathing the breath of the monster   combining agile and context-driven
Breathing the breath of the monster combining agile and context-drivenIlari Henrik Aegerter
 
Brightfind world usability day 2016 full deck final
Brightfind world usability day 2016   full deck finalBrightfind world usability day 2016   full deck final
Brightfind world usability day 2016 full deck finalBrightfind
 
Seven Peaks Speaks: User testing to action
Seven Peaks Speaks: User testing to actionSeven Peaks Speaks: User testing to action
Seven Peaks Speaks: User testing to actionSeven Peaks Speaks
 
SentimentAnalysisofTwitterProductReviewsDocument.pdf
SentimentAnalysisofTwitterProductReviewsDocument.pdfSentimentAnalysisofTwitterProductReviewsDocument.pdf
SentimentAnalysisofTwitterProductReviewsDocument.pdfDevinSohi
 

Similar to Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys (20)

Bba q&a study final white
Bba q&a study final whiteBba q&a study final white
Bba q&a study final white
 
Growth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - AntwerpGrowth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - Antwerp
 
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
 
How to Correctly Use Experimentation in PM by Google PM
How to Correctly Use Experimentation in PM by Google PMHow to Correctly Use Experimentation in PM by Google PM
How to Correctly Use Experimentation in PM by Google PM
 
DIT Digitial Marketing Forum: Analytics
DIT Digitial Marketing Forum: AnalyticsDIT Digitial Marketing Forum: Analytics
DIT Digitial Marketing Forum: Analytics
 
Creating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvementCreating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvement
 
A Happy Marriage between Context-Driven and Agile
A Happy Marriage between Context-Driven and AgileA Happy Marriage between Context-Driven and Agile
A Happy Marriage between Context-Driven and Agile
 
JDO 2019: Data Science for Developers - Matthew Renze
JDO 2019: Data Science for Developers -  Matthew RenzeJDO 2019: Data Science for Developers -  Matthew Renze
JDO 2019: Data Science for Developers - Matthew Renze
 
08-Task Analysis.pdf
08-Task Analysis.pdf08-Task Analysis.pdf
08-Task Analysis.pdf
 
Usability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working GroupUsability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working Group
 
Intuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation ProgramIntuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation Program
 
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
 
The Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMThe Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PM
 
Rinse and Repeat : The Spiral of Applied Machine Learning
Rinse and Repeat : The Spiral of Applied Machine LearningRinse and Repeat : The Spiral of Applied Machine Learning
Rinse and Repeat : The Spiral of Applied Machine Learning
 
Understand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakesUnderstand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakes
 
Breathing the breath of the monster combining agile and context-driven
Breathing the breath of the monster   combining agile and context-drivenBreathing the breath of the monster   combining agile and context-driven
Breathing the breath of the monster combining agile and context-driven
 
Brightfind world usability day 2016 full deck final
Brightfind world usability day 2016   full deck finalBrightfind world usability day 2016   full deck final
Brightfind world usability day 2016 full deck final
 
Seven Peaks Speaks: User testing to action
Seven Peaks Speaks: User testing to actionSeven Peaks Speaks: User testing to action
Seven Peaks Speaks: User testing to action
 
INTRODUCTION
INTRODUCTIONINTRODUCTION
INTRODUCTION
 
SentimentAnalysisofTwitterProductReviewsDocument.pdf
SentimentAnalysisofTwitterProductReviewsDocument.pdfSentimentAnalysisofTwitterProductReviewsDocument.pdf
SentimentAnalysisofTwitterProductReviewsDocument.pdf
 

More from Ray Poynter

The State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and FindingsThe State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and FindingsRay Poynter
 
ResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis toolResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis toolRay Poynter
 
AI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from YasnaAI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from YasnaRay Poynter
 
Artificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So FarArtificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So FarRay Poynter
 
State of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMRState of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMRRay Poynter
 
Sudden Death of Beliefs
Sudden Death of BeliefsSudden Death of Beliefs
Sudden Death of BeliefsRay Poynter
 
Uncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden NarrativesUncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden NarrativesRay Poynter
 
Narrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at MondelēzNarrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at MondelēzRay Poynter
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in FocusRay Poynter
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in FocusRay Poynter
 
The State of Insights – September 2023
The State of Insights – September 2023The State of Insights – September 2023
The State of Insights – September 2023Ray Poynter
 
Research Thinking in the age of AI
Research Thinking in the age of AIResearch Thinking in the age of AI
Research Thinking in the age of AIRay Poynter
 
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...Ray Poynter
 
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...Ray Poynter
 
Using Word Cloud Plus to Code Open Ended Text
Using Word Cloud Plus to Code Open Ended TextUsing Word Cloud Plus to Code Open Ended Text
Using Word Cloud Plus to Code Open Ended TextRay Poynter
 
ChatGPT and Market Research: the first 100 days!
ChatGPT and Market Research: the first 100 days!ChatGPT and Market Research: the first 100 days!
ChatGPT and Market Research: the first 100 days!Ray Poynter
 
Word Cloud Plus with Will and Ray Poynter
Word Cloud Plus with Will and Ray PoynterWord Cloud Plus with Will and Ray Poynter
Word Cloud Plus with Will and Ray PoynterRay Poynter
 
Curious Humans: the lost key to impactful insights
Curious Humans: the lost key to impactful insightsCurious Humans: the lost key to impactful insights
Curious Humans: the lost key to impactful insightsRay Poynter
 
Research Thinking – Your career Superpower!
Research Thinking – Your career Superpower!Research Thinking – Your career Superpower!
Research Thinking – Your career Superpower!Ray Poynter
 
Insights & Market Research in 2027 – Scenario Thinking
Insights & Market Research in 2027 – Scenario ThinkingInsights & Market Research in 2027 – Scenario Thinking
Insights & Market Research in 2027 – Scenario ThinkingRay Poynter
 

More from Ray Poynter (20)

The State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and FindingsThe State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and Findings
 
ResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis toolResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis tool
 
AI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from YasnaAI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from Yasna
 
Artificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So FarArtificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So Far
 
State of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMRState of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMR
 
Sudden Death of Beliefs
Sudden Death of BeliefsSudden Death of Beliefs
Sudden Death of Beliefs
 
Uncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden NarrativesUncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden Narratives
 
Narrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at MondelēzNarrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at Mondelēz
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
 
The State of Insights – September 2023
The State of Insights – September 2023The State of Insights – September 2023
The State of Insights – September 2023
 
Research Thinking in the age of AI
Research Thinking in the age of AIResearch Thinking in the age of AI
Research Thinking in the age of AI
 
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
 
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
 
Using Word Cloud Plus to Code Open Ended Text
Using Word Cloud Plus to Code Open Ended TextUsing Word Cloud Plus to Code Open Ended Text
Using Word Cloud Plus to Code Open Ended Text
 
ChatGPT and Market Research: the first 100 days!
ChatGPT and Market Research: the first 100 days!ChatGPT and Market Research: the first 100 days!
ChatGPT and Market Research: the first 100 days!
 
Word Cloud Plus with Will and Ray Poynter
Word Cloud Plus with Will and Ray PoynterWord Cloud Plus with Will and Ray Poynter
Word Cloud Plus with Will and Ray Poynter
 
Curious Humans: the lost key to impactful insights
Curious Humans: the lost key to impactful insightsCurious Humans: the lost key to impactful insights
Curious Humans: the lost key to impactful insights
 
Research Thinking – Your career Superpower!
Research Thinking – Your career Superpower!Research Thinking – Your career Superpower!
Research Thinking – Your career Superpower!
 
Insights & Market Research in 2027 – Scenario Thinking
Insights & Market Research in 2027 – Scenario ThinkingInsights & Market Research in 2027 – Scenario Thinking
Insights & Market Research in 2027 – Scenario Thinking
 

Recently uploaded

ICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfVanessa Camilleri
 
Using Grammatical Signals Suitable to Patterns of Idea Development
Using Grammatical Signals Suitable to Patterns of Idea DevelopmentUsing Grammatical Signals Suitable to Patterns of Idea Development
Using Grammatical Signals Suitable to Patterns of Idea Developmentchesterberbo7
 
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfInclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfTechSoup
 
Expanded definition: technical and operational
Expanded definition: technical and operationalExpanded definition: technical and operational
Expanded definition: technical and operationalssuser3e220a
 
Concurrency Control in Database Management system
Concurrency Control in Database Management systemConcurrency Control in Database Management system
Concurrency Control in Database Management systemChristalin Nelson
 
Grade 9 Quarter 4 Dll Grade 9 Quarter 4 DLL.pdf
Grade 9 Quarter 4 Dll Grade 9 Quarter 4 DLL.pdfGrade 9 Quarter 4 Dll Grade 9 Quarter 4 DLL.pdf
Grade 9 Quarter 4 Dll Grade 9 Quarter 4 DLL.pdfJemuel Francisco
 
Oppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmOppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmStan Meyer
 
Textual Evidence in Reading and Writing of SHS
Textual Evidence in Reading and Writing of SHSTextual Evidence in Reading and Writing of SHS
Textual Evidence in Reading and Writing of SHSMae Pangan
 
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxINTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxHumphrey A Beña
 
ROLES IN A STAGE PRODUCTION in arts.pptx
ROLES IN A STAGE PRODUCTION in arts.pptxROLES IN A STAGE PRODUCTION in arts.pptx
ROLES IN A STAGE PRODUCTION in arts.pptxVanesaIglesias10
 
4.16.24 Poverty and Precarity--Desmond.pptx
4.16.24 Poverty and Precarity--Desmond.pptx4.16.24 Poverty and Precarity--Desmond.pptx
4.16.24 Poverty and Precarity--Desmond.pptxmary850239
 
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONTHEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONHumphrey A Beña
 
Virtual-Orientation-on-the-Administration-of-NATG12-NATG6-and-ELLNA.pdf
Virtual-Orientation-on-the-Administration-of-NATG12-NATG6-and-ELLNA.pdfVirtual-Orientation-on-the-Administration-of-NATG12-NATG6-and-ELLNA.pdf
Virtual-Orientation-on-the-Administration-of-NATG12-NATG6-and-ELLNA.pdfErwinPantujan2
 
CONCEPT OF MUTATION AND ITS CLASSIFICATION .pptx
CONCEPT OF MUTATION AND ITS CLASSIFICATION .pptxCONCEPT OF MUTATION AND ITS CLASSIFICATION .pptx
CONCEPT OF MUTATION AND ITS CLASSIFICATION .pptxAnupkumar Sharma
 
Choosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for ParentsChoosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for Parentsnavabharathschool99
 
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)lakshayb543
 

Recently uploaded (20)

ICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdf
 
Using Grammatical Signals Suitable to Patterns of Idea Development
Using Grammatical Signals Suitable to Patterns of Idea DevelopmentUsing Grammatical Signals Suitable to Patterns of Idea Development
Using Grammatical Signals Suitable to Patterns of Idea Development
 
Mattingly "AI & Prompt Design: Large Language Models"
Mattingly "AI & Prompt Design: Large Language Models"Mattingly "AI & Prompt Design: Large Language Models"
Mattingly "AI & Prompt Design: Large Language Models"
 
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfInclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
 
Paradigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTAParadigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTA
 
Expanded definition: technical and operational
Expanded definition: technical and operationalExpanded definition: technical and operational
Expanded definition: technical and operational
 
Concurrency Control in Database Management system
Concurrency Control in Database Management systemConcurrency Control in Database Management system
Concurrency Control in Database Management system
 
Grade 9 Quarter 4 Dll Grade 9 Quarter 4 DLL.pdf
Grade 9 Quarter 4 Dll Grade 9 Quarter 4 DLL.pdfGrade 9 Quarter 4 Dll Grade 9 Quarter 4 DLL.pdf
Grade 9 Quarter 4 Dll Grade 9 Quarter 4 DLL.pdf
 
Oppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmOppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and Film
 
Textual Evidence in Reading and Writing of SHS
Textual Evidence in Reading and Writing of SHSTextual Evidence in Reading and Writing of SHS
Textual Evidence in Reading and Writing of SHS
 
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxINTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
 
ROLES IN A STAGE PRODUCTION in arts.pptx
ROLES IN A STAGE PRODUCTION in arts.pptxROLES IN A STAGE PRODUCTION in arts.pptx
ROLES IN A STAGE PRODUCTION in arts.pptx
 
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptxYOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
 
4.16.24 Poverty and Precarity--Desmond.pptx
4.16.24 Poverty and Precarity--Desmond.pptx4.16.24 Poverty and Precarity--Desmond.pptx
4.16.24 Poverty and Precarity--Desmond.pptx
 
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONTHEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
 
LEFT_ON_C'N_ PRELIMS_EL_DORADO_2024.pptx
LEFT_ON_C'N_ PRELIMS_EL_DORADO_2024.pptxLEFT_ON_C'N_ PRELIMS_EL_DORADO_2024.pptx
LEFT_ON_C'N_ PRELIMS_EL_DORADO_2024.pptx
 
Virtual-Orientation-on-the-Administration-of-NATG12-NATG6-and-ELLNA.pdf
Virtual-Orientation-on-the-Administration-of-NATG12-NATG6-and-ELLNA.pdfVirtual-Orientation-on-the-Administration-of-NATG12-NATG6-and-ELLNA.pdf
Virtual-Orientation-on-the-Administration-of-NATG12-NATG6-and-ELLNA.pdf
 
CONCEPT OF MUTATION AND ITS CLASSIFICATION .pptx
CONCEPT OF MUTATION AND ITS CLASSIFICATION .pptxCONCEPT OF MUTATION AND ITS CLASSIFICATION .pptx
CONCEPT OF MUTATION AND ITS CLASSIFICATION .pptx
 
Choosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for ParentsChoosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for Parents
 
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
 

Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys

  • 1. Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys NewMR Webinar June 2023 Karine Pepin 2CV Research
  • 2. LONDON I LOS ANGELES I SAN FRANCISCO I SINGAPORE Using Generative AI to Assess the Quality of Open- Ended Responses in Surveys By Karine Pepin SVP
  • 3. 3 Current Challenges with Open-Ended Responses How Generative AI can help Best practices
  • 4. 4 Checks of Data Integrity Source: Insights Association Data Integrity Initiative A Toolkit from the Insights Association In Partnership with a Council of Industry Leader TIME SPENT Pre- survey In-survey QC checks (Automated) Post-survey checks (mostly manual) o Geo-Location Tracking o Digital Device Fingerprinting o IP-Deduplication o Bots o Bot Script o Click-Farm o Form Filler Check o Instructional Manipulation Check o Low Incidence Check o Speeder/Racer o Straightlining o Red Herring/Trap Question o Contradictory Answers o Open-End Response Validation Survey o Data Discrepancy Checks o Open-End Response Review o Benchmark Comparison o MaxDiff Question Check
  • 5. 5 In-survey validation: What it does Type of issues flagged in survey • Copy and paste • Profanity • Proper grammar • Duplicate response • Gibberish Be D add is do ha sh he be sh he be can ha DJ ha sh ha sh ha can be sh vs CZ sh ha en be xxx Amazon all very good and very nice debit cards in very nice and very good information technology they proide beat service in short best of time because I feel like you are doing well da*n I have no problem getting to the money 5
  • 6. 6 In-survey validation: What it doesn’t do Thinking back to the last purchase you made using this type of payment, where was the purchase made and why did you choose this specific form of payment? In his inaugural address, Donald J. Trump maintained the “Make America Great Again” campaign mantra that helped sweep him into office. And he took the theme of “transfer of power” a step further, saying “we are transferring power from Washington I’m just making More money and I’m Saving up even more No thanks I don’t want to answer this question again I just got a call from the lord and I am in your high school. Tesla company is growing very fast and they make the best electric vehicles in the United States. Very much I appreciate you feeling today I am Good night too much for your help today to get a good one to come over and watch
  • 7. 7 Open-Ended Response: What we need OpenAI's natural language processing models are designed to understand the context in which language is used allowing them to analyze the meaning and intent behind open-ended responses.
  • 8. 8 Here’s a simple example using ChatGPT
  • 9. 9 Limitations with ChatGPT Not possible to provide additional training data to ChatGPT directly Using GPT via Google Apps Script solves these challenges Cannot be integrated directly into Google Sheets/ Excel which allows you allows you to analyze more than one open-ended at the time making it hard to work with large datasets. Doesn’t get smarter: Doesn’t scale
  • 10. 10 Automation Process Automate Use Google Apps Script to build a custom VERIFY_RESPONSE() formula that calls the OpenAI Chat completion API to receive a YES/NO assessment along with a reason Train Provide open-ended responses as training data to help the model learn to generate a more accurate assessment to similar open-ended responses Iterate Run open-ended responses in batches in order improve on your training
  • 12. 12 Google Sheet Set-Up: Training Data
  • 13. 13 Google Sheet Set-Up: Survey Question
  • 14. 14 Google Sheet Set-Up: Open-Ended Response Input/Output
  • 15. 15 Google Sheet Set-Up: Open-Ended Response Input/Output
  • 16. 16 Google Sheet Set-Up: Open-Ended Response Input/Output
  • 17. 17 Google Sheet Set-Up: Double API Call for Other Languages
  • 18. 18 How Does it Perform Is it better than a human? No. Is it better than doing nothing Yes! Is it faster than a human? Yes!
  • 20. 20 Best Practices Use generic OE questions designed for QC only Data Confidentiality Create a bank of OE questions so the model doesn’t have to be train each time Efficiency Use more than one OE questions to make the final assessment Accuracy Provide additional training data when new countries, languages, or audiences are added Tuning Review rejected responses carefully. Spot check accepted responses. Validation The API could be integrated directly into the survey script, but exercise caution! Survey Integration
  • 22. Q & A Ray Poynter NewMR Karine Pepin 2CV Research