The document discusses using generative AI to assess the quality of open-ended survey responses. It outlines current challenges with open-ended responses, how generative AI can help through natural language processing, and best practices for implementation. Specifically, it provides an example of using ChatGPT to analyze sample open-ended responses and discusses setting up a Google Sheet to automate the process at scale and continuously improve the model through iterative training.

1. Using Generative AI to Assess the Quality
of Open-Ended Responses in Surveys
NewMR Webinar June 2023
Karine Pepin
2CV Research

2. LONDON I LOS ANGELES I SAN FRANCISCO I SINGAPORE
Using Generative
AI to Assess the
Quality of Open-
Ended Responses
in Surveys
By Karine Pepin
SVP

3. 3
Current Challenges
with Open-Ended
Responses
How Generative AI
can help
Best practices

4. 4
Checks of Data Integrity
Source: Insights Association Data Integrity Initiative A Toolkit from the Insights Association In
Partnership with a Council of Industry Leader
TIME
SPENT
Pre-
survey
In-survey QC
checks
(Automated)
Post-survey checks
(mostly manual)
o Geo-Location Tracking
o Digital Device Fingerprinting
o IP-Deduplication
o Bots
o Bot Script
o Click-Farm
o Form Filler Check
o Instructional Manipulation Check
o Low Incidence Check
o Speeder/Racer
o Straightlining
o Red Herring/Trap Question
o Contradictory Answers
o Open-End Response Validation Survey
o Data Discrepancy Checks
o Open-End Response Review
o Benchmark Comparison
o MaxDiff Question Check

5. 5
In-survey validation:
What it does
Type of issues flagged in survey
• Copy and paste
• Profanity
• Proper grammar
• Duplicate response
• Gibberish
Be D add is do ha
sh he be sh he be
can ha DJ ha sh ha
sh ha can be sh vs
CZ sh ha en be xxx
Amazon all very
good and very nice
debit cards in very
nice and very good
information
technology
they proide
beat service in
short best of
time
because I feel like
you are doing well
da*n I have no
problem getting to
the money
5

6. 6
In-survey
validation: What it
doesn’t do
Thinking back to the last
purchase you made using this
type of payment, where was
the purchase made and why
did you choose this specific
form of payment?
In his inaugural
address, Donald J.
Trump maintained
the “Make America
Great Again”
campaign mantra
that helped sweep
him into office. And
he took the theme of
“transfer of power” a
step further, saying
“we are transferring
power from
Washington
I’m just
making
More
money and
I’m Saving
up even
more
No thanks I don’t
want to answer
this question
again I just got a
call from the lord
and I am in your
high school.
Tesla company is
growing very fast
and they make the
best electric vehicles
in the United States.
Very much I
appreciate
you feeling
today I am
Good night
too much for
your help
today to get a
good one to
come over
and watch

7. 7
Open-Ended Response: What we need
OpenAI's natural language processing models are designed to understand the context in which language is used
allowing them to analyze the meaning and intent behind open-ended responses.

8. 8
Here’s a simple example using ChatGPT

9. 9
Limitations with ChatGPT
Not possible to provide additional
training data to ChatGPT directly
Using GPT via Google Apps Script solves these challenges
Cannot be integrated directly into
Google Sheets/ Excel which allows
you allows you to analyze more than
one open-ended at the time making
it hard to work with large datasets.
Doesn’t get smarter: Doesn’t scale

10. 10
Automation Process
Automate
Use Google Apps Script to build
a custom VERIFY_RESPONSE()
formula that calls the OpenAI
Chat completion API to receive
a YES/NO assessment along
with a reason
Train
Provide open-ended responses
as training data to help the
model learn to generate a more
accurate assessment to similar
open-ended responses
Iterate
Run open-ended
responses in
batches in order
improve on your
training

11. 11
Google Sheet Set-Up: Settings

12. 12
Google Sheet Set-Up: Training Data

13. 13
Google Sheet Set-Up: Survey Question

14. 14
Google Sheet Set-Up: Open-Ended Response Input/Output

15. 15
Google Sheet Set-Up: Open-Ended Response Input/Output

16. 16
Google Sheet Set-Up: Open-Ended Response Input/Output

17. 17
Google Sheet Set-Up: Double API Call for Other Languages

18. 18
How Does it Perform
Is it better than
a human? No.
Is it better
than doing
nothing Yes!
Is it faster
than a
human? Yes!

19. 19
“Training
Makes Perfect”

20. 20
Best Practices
Use generic OE
questions designed
for QC only
Data
Confidentiality
Create a bank of OE
questions so the
model doesn’t have
to be train each time
Efficiency
Use more than one
OE questions to
make the final
assessment
Accuracy
Provide additional
training data when
new countries,
languages, or
audiences are added
Tuning
Review rejected
responses carefully.
Spot check accepted
responses.
Validation
The API could be
integrated directly
into the survey
script, but exercise
caution!
Survey
Integration

21. Thank you
Karine.Pepin@2cv.com

22. Q & A
Ray Poynter
NewMR
Karine Pepin
2CV Research

23. Sponsors