SlideShare a Scribd company logo
Using Generative AI to Assess the Quality
of Open-Ended Responses in Surveys
NewMR Webinar June 2023
Karine Pepin
2CV Research
LONDON I LOS ANGELES I SAN FRANCISCO I SINGAPORE
Using Generative
AI to Assess the
Quality of Open-
Ended Responses
in Surveys
By Karine Pepin
SVP
3
Current Challenges
with Open-Ended
Responses
How Generative AI
can help
Best practices
4
Checks of Data Integrity
Source: Insights Association Data Integrity Initiative A Toolkit from the Insights Association In
Partnership with a Council of Industry Leader
TIME
SPENT
Pre-
survey
In-survey QC
checks
(Automated)
Post-survey checks
(mostly manual)
o Geo-Location Tracking
o Digital Device Fingerprinting
o IP-Deduplication
o Bots
o Bot Script
o Click-Farm
o Form Filler Check
o Instructional Manipulation Check
o Low Incidence Check
o Speeder/Racer
o Straightlining
o Red Herring/Trap Question
o Contradictory Answers
o Open-End Response Validation Survey
o Data Discrepancy Checks
o Open-End Response Review
o Benchmark Comparison
o MaxDiff Question Check
5
In-survey validation:
What it does
Type of issues flagged in survey
• Copy and paste
• Profanity
• Proper grammar
• Duplicate response
• Gibberish
Be D add is do ha
sh he be sh he be
can ha DJ ha sh ha
sh ha can be sh vs
CZ sh ha en be xxx
Amazon all very
good and very nice
debit cards in very
nice and very good
information
technology
they proide
beat service in
short best of
time
because I feel like
you are doing well
da*n I have no
problem getting to
the money
5
6
In-survey
validation: What it
doesn’t do
Thinking back to the last
purchase you made using this
type of payment, where was
the purchase made and why
did you choose this specific
form of payment?
In his inaugural
address, Donald J.
Trump maintained
the “Make America
Great Again”
campaign mantra
that helped sweep
him into office. And
he took the theme of
“transfer of power” a
step further, saying
“we are transferring
power from
Washington
I’m just
making
More
money and
I’m Saving
up even
more
No thanks I don’t
want to answer
this question
again I just got a
call from the lord
and I am in your
high school.
Tesla company is
growing very fast
and they make the
best electric vehicles
in the United States.
Very much I
appreciate
you feeling
today I am
Good night
too much for
your help
today to get a
good one to
come over
and watch
7
Open-Ended Response: What we need
OpenAI's natural language processing models are designed to understand the context in which language is used
allowing them to analyze the meaning and intent behind open-ended responses.
8
Here’s a simple example using ChatGPT
9
Limitations with ChatGPT
Not possible to provide additional
training data to ChatGPT directly
Using GPT via Google Apps Script solves these challenges
Cannot be integrated directly into
Google Sheets/ Excel which allows
you allows you to analyze more than
one open-ended at the time making
it hard to work with large datasets.
Doesn’t get smarter: Doesn’t scale
10
Automation Process
Automate
Use Google Apps Script to build
a custom VERIFY_RESPONSE()
formula that calls the OpenAI
Chat completion API to receive
a YES/NO assessment along
with a reason
Train
Provide open-ended responses
as training data to help the
model learn to generate a more
accurate assessment to similar
open-ended responses
Iterate
Run open-ended
responses in
batches in order
improve on your
training
11
Google Sheet Set-Up: Settings
12
Google Sheet Set-Up: Training Data
13
Google Sheet Set-Up: Survey Question
14
Google Sheet Set-Up: Open-Ended Response Input/Output
15
Google Sheet Set-Up: Open-Ended Response Input/Output
16
Google Sheet Set-Up: Open-Ended Response Input/Output
17
Google Sheet Set-Up: Double API Call for Other Languages
18
How Does it Perform
Is it better than
a human? No.
Is it better
than doing
nothing Yes!
Is it faster
than a
human? Yes!
19
“Training
Makes Perfect”
20
Best Practices
Use generic OE
questions designed
for QC only
Data
Confidentiality
Create a bank of OE
questions so the
model doesn’t have
to be train each time
Efficiency
Use more than one
OE questions to
make the final
assessment
Accuracy
Provide additional
training data when
new countries,
languages, or
audiences are added
Tuning
Review rejected
responses carefully.
Spot check accepted
responses.
Validation
The API could be
integrated directly
into the survey
script, but exercise
caution!
Survey
Integration
Thank you
Karine.Pepin@2cv.com
Q & A
Ray Poynter
NewMR
Karine Pepin
2CV Research
Sponsors

More Related Content

What's hot

LLM App Hacking (AVTOKYO2023)
LLM App Hacking (AVTOKYO2023)LLM App Hacking (AVTOKYO2023)
LLM App Hacking (AVTOKYO2023)
Shota Shinogi
 
Generative AI at the edge.pdf
Generative AI at the edge.pdfGenerative AI at the edge.pdf
Generative AI at the edge.pdf
Qualcomm Research
 
Introduction to LLMs
Introduction to LLMsIntroduction to LLMs
Introduction to LLMs
Loic Merckel
 
Large Language Models, Data & APIs - Integrating Generative AI Power into you...
Large Language Models, Data & APIs - Integrating Generative AI Power into you...Large Language Models, Data & APIs - Integrating Generative AI Power into you...
Large Language Models, Data & APIs - Integrating Generative AI Power into you...
NETUserGroupBern
 
Content In The Age of AI
Content In The Age of AIContent In The Age of AI
Content In The Age of AI
S&G Content Marketing
 
Unlocking the Power of Generative AI An Executive's Guide.pdf
Unlocking the Power of Generative AI An Executive's Guide.pdfUnlocking the Power of Generative AI An Executive's Guide.pdf
Unlocking the Power of Generative AI An Executive's Guide.pdf
PremNaraindas1
 
Jeff Dean at AI Frontiers: Trends and Developments in Deep Learning Research
Jeff Dean at AI Frontiers: Trends and Developments in Deep Learning ResearchJeff Dean at AI Frontiers: Trends and Developments in Deep Learning Research
Jeff Dean at AI Frontiers: Trends and Developments in Deep Learning Research
AI Frontiers
 
Generative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdfGenerative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdf
Liming Zhu
 
How to Interview a Data Scientist
How to Interview a Data ScientistHow to Interview a Data Scientist
How to Interview a Data Scientist
Daniel Tunkelang
 
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Alain Goudey
 
LanGCHAIN Framework
LanGCHAIN FrameworkLanGCHAIN Framework
LanGCHAIN Framework
Keymate.AI
 
Let's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchersLet's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchers
Steven Van Vaerenbergh
 
Breaking down the AI magic of ChatGPT: A technologist's lens to its powerful ...
Breaking down the AI magic of ChatGPT: A technologist's lens to its powerful ...Breaking down the AI magic of ChatGPT: A technologist's lens to its powerful ...
Breaking down the AI magic of ChatGPT: A technologist's lens to its powerful ...
rahul_net
 
How do OpenAI GPT Models Work - Misconceptions and Tips for Developers
How do OpenAI GPT Models Work - Misconceptions and Tips for DevelopersHow do OpenAI GPT Models Work - Misconceptions and Tips for Developers
How do OpenAI GPT Models Work - Misconceptions and Tips for Developers
Ivo Andreev
 
LLMs_talk_March23.pdf
LLMs_talk_March23.pdfLLMs_talk_March23.pdf
LLMs_talk_March23.pdf
ChaoYang81
 
The AI Index 2023 Annual Report by Stanford University.pdf
The AI Index 2023 Annual Report by Stanford University.pdfThe AI Index 2023 Annual Report by Stanford University.pdf
The AI Index 2023 Annual Report by Stanford University.pdf
AI Geek (wishesh)
 
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Sri Ambati
 
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer RoadmapEpisode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Anant Corporation
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?
Maxim Salnikov
 
Global Azure Bootcamp Pune 2023 - Lead the AI era with Microsoft Azure.pdf
Global Azure Bootcamp Pune 2023 -  Lead the AI era with Microsoft Azure.pdfGlobal Azure Bootcamp Pune 2023 -  Lead the AI era with Microsoft Azure.pdf
Global Azure Bootcamp Pune 2023 - Lead the AI era with Microsoft Azure.pdf
Aroh Shukla
 

What's hot (20)

LLM App Hacking (AVTOKYO2023)
LLM App Hacking (AVTOKYO2023)LLM App Hacking (AVTOKYO2023)
LLM App Hacking (AVTOKYO2023)
 
Generative AI at the edge.pdf
Generative AI at the edge.pdfGenerative AI at the edge.pdf
Generative AI at the edge.pdf
 
Introduction to LLMs
Introduction to LLMsIntroduction to LLMs
Introduction to LLMs
 
Large Language Models, Data & APIs - Integrating Generative AI Power into you...
Large Language Models, Data & APIs - Integrating Generative AI Power into you...Large Language Models, Data & APIs - Integrating Generative AI Power into you...
Large Language Models, Data & APIs - Integrating Generative AI Power into you...
 
Content In The Age of AI
Content In The Age of AIContent In The Age of AI
Content In The Age of AI
 
Unlocking the Power of Generative AI An Executive's Guide.pdf
Unlocking the Power of Generative AI An Executive's Guide.pdfUnlocking the Power of Generative AI An Executive's Guide.pdf
Unlocking the Power of Generative AI An Executive's Guide.pdf
 
Jeff Dean at AI Frontiers: Trends and Developments in Deep Learning Research
Jeff Dean at AI Frontiers: Trends and Developments in Deep Learning ResearchJeff Dean at AI Frontiers: Trends and Developments in Deep Learning Research
Jeff Dean at AI Frontiers: Trends and Developments in Deep Learning Research
 
Generative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdfGenerative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdf
 
How to Interview a Data Scientist
How to Interview a Data ScientistHow to Interview a Data Scientist
How to Interview a Data Scientist
 
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
Chat GPT and Generative AI in Higher Education - Empowering Educators and Lea...
 
LanGCHAIN Framework
LanGCHAIN FrameworkLanGCHAIN Framework
LanGCHAIN Framework
 
Let's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchersLet's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchers
 
Breaking down the AI magic of ChatGPT: A technologist's lens to its powerful ...
Breaking down the AI magic of ChatGPT: A technologist's lens to its powerful ...Breaking down the AI magic of ChatGPT: A technologist's lens to its powerful ...
Breaking down the AI magic of ChatGPT: A technologist's lens to its powerful ...
 
How do OpenAI GPT Models Work - Misconceptions and Tips for Developers
How do OpenAI GPT Models Work - Misconceptions and Tips for DevelopersHow do OpenAI GPT Models Work - Misconceptions and Tips for Developers
How do OpenAI GPT Models Work - Misconceptions and Tips for Developers
 
LLMs_talk_March23.pdf
LLMs_talk_March23.pdfLLMs_talk_March23.pdf
LLMs_talk_March23.pdf
 
The AI Index 2023 Annual Report by Stanford University.pdf
The AI Index 2023 Annual Report by Stanford University.pdfThe AI Index 2023 Annual Report by Stanford University.pdf
The AI Index 2023 Annual Report by Stanford University.pdf
 
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
 
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer RoadmapEpisode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?
 
Global Azure Bootcamp Pune 2023 - Lead the AI era with Microsoft Azure.pdf
Global Azure Bootcamp Pune 2023 -  Lead the AI era with Microsoft Azure.pdfGlobal Azure Bootcamp Pune 2023 -  Lead the AI era with Microsoft Azure.pdf
Global Azure Bootcamp Pune 2023 - Lead the AI era with Microsoft Azure.pdf
 

Similar to Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys

Bba q&a study final white
Bba q&a study final whiteBba q&a study final white
Bba q&a study final white
Greg Sterling
 
Growth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - AntwerpGrowth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - Antwerp
Thibault Imbert
 
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
DataScienceConferenc1
 
How to Correctly Use Experimentation in PM by Google PM
How to Correctly Use Experimentation in PM by Google PMHow to Correctly Use Experimentation in PM by Google PM
How to Correctly Use Experimentation in PM by Google PM
Product School
 
DIT Digitial Marketing Forum: Analytics
DIT Digitial Marketing Forum: AnalyticsDIT Digitial Marketing Forum: Analytics
DIT Digitial Marketing Forum: Analytics
Lar Veale
 
Creating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvementCreating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvement
Ben Dressler
 
A Happy Marriage between Context-Driven and Agile
A Happy Marriage between Context-Driven and AgileA Happy Marriage between Context-Driven and Agile
A Happy Marriage between Context-Driven and Agile
Ilari Henrik Aegerter
 
JDO 2019: Data Science for Developers - Matthew Renze
JDO 2019: Data Science for Developers -  Matthew RenzeJDO 2019: Data Science for Developers -  Matthew Renze
JDO 2019: Data Science for Developers - Matthew Renze
PROIDEA
 
08-Task Analysis.pdf
08-Task Analysis.pdf08-Task Analysis.pdf
08-Task Analysis.pdf
shreyakale11
 
Usability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working GroupUsability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working Group
NormanMendoza
 
Intuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation ProgramIntuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation Program
Optimizely
 
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Grid Dynamics
 
The Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMThe Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PM
Product School
 
Rinse and Repeat : The Spiral of Applied Machine Learning
Rinse and Repeat : The Spiral of Applied Machine LearningRinse and Repeat : The Spiral of Applied Machine Learning
Rinse and Repeat : The Spiral of Applied Machine Learning
Anna Chaney
 
Understand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakesUnderstand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakes
TheFamily
 
Breathing the breath of the monster combining agile and context-driven
Breathing the breath of the monster   combining agile and context-drivenBreathing the breath of the monster   combining agile and context-driven
Breathing the breath of the monster combining agile and context-driven
Ilari Henrik Aegerter
 
Brightfind world usability day 2016 full deck final
Brightfind world usability day 2016   full deck finalBrightfind world usability day 2016   full deck final
Brightfind world usability day 2016 full deck final
Brightfind
 
Seven Peaks Speaks: User testing to action
Seven Peaks Speaks: User testing to actionSeven Peaks Speaks: User testing to action
Seven Peaks Speaks: User testing to action
Seven Peaks Speaks
 
INTRODUCTION
INTRODUCTIONINTRODUCTION
INTRODUCTION
Salman Mohd
 
SentimentAnalysisofTwitterProductReviewsDocument.pdf
SentimentAnalysisofTwitterProductReviewsDocument.pdfSentimentAnalysisofTwitterProductReviewsDocument.pdf
SentimentAnalysisofTwitterProductReviewsDocument.pdf
DevinSohi
 

Similar to Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys (20)

Bba q&a study final white
Bba q&a study final whiteBba q&a study final white
Bba q&a study final white
 
Growth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - AntwerpGrowth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - Antwerp
 
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
[DSC Europe 22] Avoid mistakes building AI products - Karol Przystalski
 
How to Correctly Use Experimentation in PM by Google PM
How to Correctly Use Experimentation in PM by Google PMHow to Correctly Use Experimentation in PM by Google PM
How to Correctly Use Experimentation in PM by Google PM
 
DIT Digitial Marketing Forum: Analytics
DIT Digitial Marketing Forum: AnalyticsDIT Digitial Marketing Forum: Analytics
DIT Digitial Marketing Forum: Analytics
 
Creating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvementCreating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvement
 
A Happy Marriage between Context-Driven and Agile
A Happy Marriage between Context-Driven and AgileA Happy Marriage between Context-Driven and Agile
A Happy Marriage between Context-Driven and Agile
 
JDO 2019: Data Science for Developers - Matthew Renze
JDO 2019: Data Science for Developers -  Matthew RenzeJDO 2019: Data Science for Developers -  Matthew Renze
JDO 2019: Data Science for Developers - Matthew Renze
 
08-Task Analysis.pdf
08-Task Analysis.pdf08-Task Analysis.pdf
08-Task Analysis.pdf
 
Usability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working GroupUsability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working Group
 
Intuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation ProgramIntuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation Program
 
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
Dialogflow Chat Experiences Best Practices for Intent Detection // Measuring ...
 
The Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMThe Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PM
 
Rinse and Repeat : The Spiral of Applied Machine Learning
Rinse and Repeat : The Spiral of Applied Machine LearningRinse and Repeat : The Spiral of Applied Machine Learning
Rinse and Repeat : The Spiral of Applied Machine Learning
 
Understand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakesUnderstand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakes
 
Breathing the breath of the monster combining agile and context-driven
Breathing the breath of the monster   combining agile and context-drivenBreathing the breath of the monster   combining agile and context-driven
Breathing the breath of the monster combining agile and context-driven
 
Brightfind world usability day 2016 full deck final
Brightfind world usability day 2016   full deck finalBrightfind world usability day 2016   full deck final
Brightfind world usability day 2016 full deck final
 
Seven Peaks Speaks: User testing to action
Seven Peaks Speaks: User testing to actionSeven Peaks Speaks: User testing to action
Seven Peaks Speaks: User testing to action
 
INTRODUCTION
INTRODUCTIONINTRODUCTION
INTRODUCTION
 
SentimentAnalysisofTwitterProductReviewsDocument.pdf
SentimentAnalysisofTwitterProductReviewsDocument.pdfSentimentAnalysisofTwitterProductReviewsDocument.pdf
SentimentAnalysisofTwitterProductReviewsDocument.pdf
 

More from Ray Poynter

The State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and FindingsThe State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and Findings
Ray Poynter
 
ResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis toolResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis tool
Ray Poynter
 
AI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from YasnaAI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from Yasna
Ray Poynter
 
Artificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So FarArtificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So Far
Ray Poynter
 
State of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMRState of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMR
Ray Poynter
 
Sudden Death of Beliefs
Sudden Death of BeliefsSudden Death of Beliefs
Sudden Death of Beliefs
Ray Poynter
 
Uncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden NarrativesUncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden Narratives
Ray Poynter
 
Narrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at MondelēzNarrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at Mondelēz
Ray Poynter
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
Ray Poynter
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
Ray Poynter
 
The State of Insights – September 2023
The State of Insights – September 2023The State of Insights – September 2023
The State of Insights – September 2023
Ray Poynter
 
Research Thinking in the age of AI
Research Thinking in the age of AIResearch Thinking in the age of AI
Research Thinking in the age of AI
Ray Poynter
 
How might AI impact Research and Insights over the next two years?
How might AI impact Research and Insights over the next two years?How might AI impact Research and Insights over the next two years?
How might AI impact Research and Insights over the next two years?
Ray Poynter
 
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
Ray Poynter
 
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
Ray Poynter
 
Using Word Cloud Plus to Code Open Ended Text
Using Word Cloud Plus to Code Open Ended TextUsing Word Cloud Plus to Code Open Ended Text
Using Word Cloud Plus to Code Open Ended Text
Ray Poynter
 
ChatGPT and Market Research: the first 100 days!
ChatGPT and Market Research: the first 100 days!ChatGPT and Market Research: the first 100 days!
ChatGPT and Market Research: the first 100 days!
Ray Poynter
 
Word Cloud Plus with Will and Ray Poynter
Word Cloud Plus with Will and Ray PoynterWord Cloud Plus with Will and Ray Poynter
Word Cloud Plus with Will and Ray Poynter
Ray Poynter
 
Curious Humans: the lost key to impactful insights
Curious Humans: the lost key to impactful insightsCurious Humans: the lost key to impactful insights
Curious Humans: the lost key to impactful insights
Ray Poynter
 
Research Thinking – Your career Superpower!
Research Thinking – Your career Superpower!Research Thinking – Your career Superpower!
Research Thinking – Your career Superpower!
Ray Poynter
 

More from Ray Poynter (20)

The State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and FindingsThe State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and Findings
 
ResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis toolResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis tool
 
AI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from YasnaAI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from Yasna
 
Artificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So FarArtificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So Far
 
State of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMRState of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMR
 
Sudden Death of Beliefs
Sudden Death of BeliefsSudden Death of Beliefs
Sudden Death of Beliefs
 
Uncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden NarrativesUncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden Narratives
 
Narrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at MondelēzNarrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at Mondelēz
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
 
The State of Insights – September 2023
The State of Insights – September 2023The State of Insights – September 2023
The State of Insights – September 2023
 
Research Thinking in the age of AI
Research Thinking in the age of AIResearch Thinking in the age of AI
Research Thinking in the age of AI
 
How might AI impact Research and Insights over the next two years?
How might AI impact Research and Insights over the next two years?How might AI impact Research and Insights over the next two years?
How might AI impact Research and Insights over the next two years?
 
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
 
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
 
Using Word Cloud Plus to Code Open Ended Text
Using Word Cloud Plus to Code Open Ended TextUsing Word Cloud Plus to Code Open Ended Text
Using Word Cloud Plus to Code Open Ended Text
 
ChatGPT and Market Research: the first 100 days!
ChatGPT and Market Research: the first 100 days!ChatGPT and Market Research: the first 100 days!
ChatGPT and Market Research: the first 100 days!
 
Word Cloud Plus with Will and Ray Poynter
Word Cloud Plus with Will and Ray PoynterWord Cloud Plus with Will and Ray Poynter
Word Cloud Plus with Will and Ray Poynter
 
Curious Humans: the lost key to impactful insights
Curious Humans: the lost key to impactful insightsCurious Humans: the lost key to impactful insights
Curious Humans: the lost key to impactful insights
 
Research Thinking – Your career Superpower!
Research Thinking – Your career Superpower!Research Thinking – Your career Superpower!
Research Thinking – Your career Superpower!
 

Recently uploaded

11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx
UmeshTimilsina1
 
How To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-marketHow To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-market
Sikandar Ali
 
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Alvaro Barbosa
 
Open Source and AI - ByWater Closing Keynote Presentation.pdf
Open Source and AI - ByWater Closing Keynote Presentation.pdfOpen Source and AI - ByWater Closing Keynote Presentation.pdf
Open Source and AI - ByWater Closing Keynote Presentation.pdf
Jessica Zairo
 
C# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdfC# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdf
Scholarhat
 
Dot NET Interview Questions PDF By ScholarHat
Dot NET Interview Questions PDF By ScholarHatDot NET Interview Questions PDF By ScholarHat
Dot NET Interview Questions PDF By ScholarHat
Scholarhat
 
FIRST AID PRESENTATION ON INDUSTRIAL SAFETY by dr lal.ppt
FIRST AID PRESENTATION ON INDUSTRIAL SAFETY by dr lal.pptFIRST AID PRESENTATION ON INDUSTRIAL SAFETY by dr lal.ppt
FIRST AID PRESENTATION ON INDUSTRIAL SAFETY by dr lal.ppt
ashutoshklal29
 
QCE – Unpacking the syllabus Implications for Senior School practices and ass...
QCE – Unpacking the syllabus Implications for Senior School practices and ass...QCE – Unpacking the syllabus Implications for Senior School practices and ass...
QCE – Unpacking the syllabus Implications for Senior School practices and ass...
mansk2
 
Introduction to Banking System in India.ppt
Introduction to Banking System in India.pptIntroduction to Banking System in India.ppt
Introduction to Banking System in India.ppt
Dr. S. Bulomine Regi
 
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
MANIVALANSR
 
RDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEWRDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEW
Murugan Solaiyappan
 
1. Importance_of_reducing_postharvest_loss.pptx
1. Importance_of_reducing_postharvest_loss.pptx1. Importance_of_reducing_postharvest_loss.pptx
1. Importance_of_reducing_postharvest_loss.pptx
UmeshTimilsina1
 
MVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHatMVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHat
Scholarhat
 
V2-NLC-Certificate-of-Completion_Learner.docx
V2-NLC-Certificate-of-Completion_Learner.docxV2-NLC-Certificate-of-Completion_Learner.docx
V2-NLC-Certificate-of-Completion_Learner.docx
302491
 
Introduction to Google Productivity Tools for Office and Personal Use
Introduction to Google Productivity Tools for Office and Personal UseIntroduction to Google Productivity Tools for Office and Personal Use
Introduction to Google Productivity Tools for Office and Personal Use
Excellence Foundation for South Sudan
 
React Interview Question PDF By ScholarHat
React Interview Question PDF By ScholarHatReact Interview Question PDF By ScholarHat
React Interview Question PDF By ScholarHat
Scholarhat
 
2 Post harvest Physiology of Horticulture produce.pptx
2 Post harvest Physiology of Horticulture  produce.pptx2 Post harvest Physiology of Horticulture  produce.pptx
2 Post harvest Physiology of Horticulture produce.pptx
UmeshTimilsina1
 
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdfPRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
nservice241
 
JavaScript Interview Questions PDF By ScholarHat
JavaScript Interview  Questions PDF By ScholarHatJavaScript Interview  Questions PDF By ScholarHat
JavaScript Interview Questions PDF By ScholarHat
Scholarhat
 
MATATAG CURRICULUM sample lesson exemplar.docx
MATATAG CURRICULUM sample lesson exemplar.docxMATATAG CURRICULUM sample lesson exemplar.docx
MATATAG CURRICULUM sample lesson exemplar.docx
yardenmendoza
 

Recently uploaded (20)

11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx
 
How To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-marketHow To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-market
 
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
 
Open Source and AI - ByWater Closing Keynote Presentation.pdf
Open Source and AI - ByWater Closing Keynote Presentation.pdfOpen Source and AI - ByWater Closing Keynote Presentation.pdf
Open Source and AI - ByWater Closing Keynote Presentation.pdf
 
C# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdfC# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdf
 
Dot NET Interview Questions PDF By ScholarHat
Dot NET Interview Questions PDF By ScholarHatDot NET Interview Questions PDF By ScholarHat
Dot NET Interview Questions PDF By ScholarHat
 
FIRST AID PRESENTATION ON INDUSTRIAL SAFETY by dr lal.ppt
FIRST AID PRESENTATION ON INDUSTRIAL SAFETY by dr lal.pptFIRST AID PRESENTATION ON INDUSTRIAL SAFETY by dr lal.ppt
FIRST AID PRESENTATION ON INDUSTRIAL SAFETY by dr lal.ppt
 
QCE – Unpacking the syllabus Implications for Senior School practices and ass...
QCE – Unpacking the syllabus Implications for Senior School practices and ass...QCE – Unpacking the syllabus Implications for Senior School practices and ass...
QCE – Unpacking the syllabus Implications for Senior School practices and ass...
 
Introduction to Banking System in India.ppt
Introduction to Banking System in India.pptIntroduction to Banking System in India.ppt
Introduction to Banking System in India.ppt
 
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
slidesgo-mastering-the-art-of-listening-insights-from-robin-sharma-2024070718...
 
RDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEWRDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEW
 
1. Importance_of_reducing_postharvest_loss.pptx
1. Importance_of_reducing_postharvest_loss.pptx1. Importance_of_reducing_postharvest_loss.pptx
1. Importance_of_reducing_postharvest_loss.pptx
 
MVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHatMVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHat
 
V2-NLC-Certificate-of-Completion_Learner.docx
V2-NLC-Certificate-of-Completion_Learner.docxV2-NLC-Certificate-of-Completion_Learner.docx
V2-NLC-Certificate-of-Completion_Learner.docx
 
Introduction to Google Productivity Tools for Office and Personal Use
Introduction to Google Productivity Tools for Office and Personal UseIntroduction to Google Productivity Tools for Office and Personal Use
Introduction to Google Productivity Tools for Office and Personal Use
 
React Interview Question PDF By ScholarHat
React Interview Question PDF By ScholarHatReact Interview Question PDF By ScholarHat
React Interview Question PDF By ScholarHat
 
2 Post harvest Physiology of Horticulture produce.pptx
2 Post harvest Physiology of Horticulture  produce.pptx2 Post harvest Physiology of Horticulture  produce.pptx
2 Post harvest Physiology of Horticulture produce.pptx
 
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdfPRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
 
JavaScript Interview Questions PDF By ScholarHat
JavaScript Interview  Questions PDF By ScholarHatJavaScript Interview  Questions PDF By ScholarHat
JavaScript Interview Questions PDF By ScholarHat
 
MATATAG CURRICULUM sample lesson exemplar.docx
MATATAG CURRICULUM sample lesson exemplar.docxMATATAG CURRICULUM sample lesson exemplar.docx
MATATAG CURRICULUM sample lesson exemplar.docx
 

Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys

  • 1. Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys NewMR Webinar June 2023 Karine Pepin 2CV Research
  • 2. LONDON I LOS ANGELES I SAN FRANCISCO I SINGAPORE Using Generative AI to Assess the Quality of Open- Ended Responses in Surveys By Karine Pepin SVP
  • 3. 3 Current Challenges with Open-Ended Responses How Generative AI can help Best practices
  • 4. 4 Checks of Data Integrity Source: Insights Association Data Integrity Initiative A Toolkit from the Insights Association In Partnership with a Council of Industry Leader TIME SPENT Pre- survey In-survey QC checks (Automated) Post-survey checks (mostly manual) o Geo-Location Tracking o Digital Device Fingerprinting o IP-Deduplication o Bots o Bot Script o Click-Farm o Form Filler Check o Instructional Manipulation Check o Low Incidence Check o Speeder/Racer o Straightlining o Red Herring/Trap Question o Contradictory Answers o Open-End Response Validation Survey o Data Discrepancy Checks o Open-End Response Review o Benchmark Comparison o MaxDiff Question Check
  • 5. 5 In-survey validation: What it does Type of issues flagged in survey • Copy and paste • Profanity • Proper grammar • Duplicate response • Gibberish Be D add is do ha sh he be sh he be can ha DJ ha sh ha sh ha can be sh vs CZ sh ha en be xxx Amazon all very good and very nice debit cards in very nice and very good information technology they proide beat service in short best of time because I feel like you are doing well da*n I have no problem getting to the money 5
  • 6. 6 In-survey validation: What it doesn’t do Thinking back to the last purchase you made using this type of payment, where was the purchase made and why did you choose this specific form of payment? In his inaugural address, Donald J. Trump maintained the “Make America Great Again” campaign mantra that helped sweep him into office. And he took the theme of “transfer of power” a step further, saying “we are transferring power from Washington I’m just making More money and I’m Saving up even more No thanks I don’t want to answer this question again I just got a call from the lord and I am in your high school. Tesla company is growing very fast and they make the best electric vehicles in the United States. Very much I appreciate you feeling today I am Good night too much for your help today to get a good one to come over and watch
  • 7. 7 Open-Ended Response: What we need OpenAI's natural language processing models are designed to understand the context in which language is used allowing them to analyze the meaning and intent behind open-ended responses.
  • 8. 8 Here’s a simple example using ChatGPT
  • 9. 9 Limitations with ChatGPT Not possible to provide additional training data to ChatGPT directly Using GPT via Google Apps Script solves these challenges Cannot be integrated directly into Google Sheets/ Excel which allows you allows you to analyze more than one open-ended at the time making it hard to work with large datasets. Doesn’t get smarter: Doesn’t scale
  • 10. 10 Automation Process Automate Use Google Apps Script to build a custom VERIFY_RESPONSE() formula that calls the OpenAI Chat completion API to receive a YES/NO assessment along with a reason Train Provide open-ended responses as training data to help the model learn to generate a more accurate assessment to similar open-ended responses Iterate Run open-ended responses in batches in order improve on your training
  • 12. 12 Google Sheet Set-Up: Training Data
  • 13. 13 Google Sheet Set-Up: Survey Question
  • 14. 14 Google Sheet Set-Up: Open-Ended Response Input/Output
  • 15. 15 Google Sheet Set-Up: Open-Ended Response Input/Output
  • 16. 16 Google Sheet Set-Up: Open-Ended Response Input/Output
  • 17. 17 Google Sheet Set-Up: Double API Call for Other Languages
  • 18. 18 How Does it Perform Is it better than a human? No. Is it better than doing nothing Yes! Is it faster than a human? Yes!
  • 20. 20 Best Practices Use generic OE questions designed for QC only Data Confidentiality Create a bank of OE questions so the model doesn’t have to be train each time Efficiency Use more than one OE questions to make the final assessment Accuracy Provide additional training data when new countries, languages, or audiences are added Tuning Review rejected responses carefully. Spot check accepted responses. Validation The API could be integrated directly into the survey script, but exercise caution! Survey Integration
  • 22. Q & A Ray Poynter NewMR Karine Pepin 2CV Research