Presented by Phil Sutcliffe from Next Intelligence. Open-ended answers in surveys can be disappointing, failing to deliver the insight that researchers hope for. Often the answers given are descriptive at best and uninformative or complete gibberish at worst. The advent of large language models (LLM's) and generative AI has brought great opportunity to change this. We now have the capability to use AI to probe within quantitative surveys. With the correct training of the generative AI models, probes can be returned that are relevant to what the participant has just said and the context of the original open-ended question. Further enhancements allow the researcher to probe on specific aspects of the participant’s response, in a similar way to how a moderator would in a depth discussion. No longer do researchers need to be frustrated that quant surveys only provide the ‘what’, with AI probing, you can now understand the ‘why’ as well and do this at scale and speed. Access the presentation recording via NewMR.org/Play-Again

1. Using Generative AI to bring Qualitative
Capabilities to Quantitative Surveys
NewMR Webinar June 2023
Phil Sutcliffe
Nexxt Intelligence

2. Sponsors

3. 3
Proprietary & Confidential
inquisitive
natural
conversation
agent

4. AI probes for elaboration
Programmed OE:
Can you tell me why you wouldn’t recommend our hotel?
Participant answers:
The service was disappointing
CONVERSATIONAL AI-FACILITATED RESEARCH CONVERSATIONS
inca SmartProbe AI probing:
What kind of service did you find unsatisfactory?

5. AI probes for specific reasons
Programmed OE:
What do you like about working at our company?
Participant answers:
The culture
CONVERSATIONAL AI-FACILITATED RESEARCH CONVERSATIONS
inca SmartProbe AI probing:
Could you tell me more about what you like about the culture?

6. AI won’t take gibberish for an answer
Programmed OE:
Why do you say you’d be unlikely to buy this product?
Participant answers:
hbduibjnksalncui
CONVERSATIONAL AI-FACILITATED RESEARCH CONVERSATIONS
inca SmartProbe AI probing:
I don’t think I understood. Do you mind rewording what you
said?

7. 7
Proprietary & Confidential
Please watch video recording of this
session to see the demo of inca’s AI
probing

8. 8
Proprietary & Confidential
Compared to other surveys you have done, how would you rate this survey experience?
“I feel like I am talking to a real person to
express my thoughts and I am being heard”
“This format was like having a conversation;
it was unique and fun and engaging”
CONVERSATIONAL AI PROBING CREATES A MORE ENGAGING
EXPERIENCE FOR SURVEY PARTICIPANTS

9. 9
Proprietary & Confidential
More
considered
responses
More
differentiated
findings
More
actionable
insights
Without Probing With Probing
11 sig. differences
across the 3
leaders
22 sig. differences
across the 3
leaders
3 of the 11 sig.
differences are on
specific policies
11 of the 22 sig.
differences are on
specific policies
Without Probing With Probing
Ave. word count per
participant: 5 words
Ave. word count per
participant: 14
words
Don’t know / not
sure: 19%
Don’t know / not
sure: 14%
Reference of a
political party/leader
without any
detail/sentiment: 8%
Reference of a
political party/leader
without any
detail/sentiment: 4%
Without Probing With Probing
4 specific strengths
that drive vote
intention identified
across the 3 leaders
12 specific strengths
that drive vote
intention identified
across the 3 leaders
8 specific
improvement areas
that detract vote
intention identified
across the 3 leaders
17 specific
improvement areas
that detract vote
intention identified
across the 3 leaders
We asked people what they think of the leaders of the 3 main political parties in Canada with an open-ended question. Then we
used inca AI to probe the response given. We then asked voting intention. We coded the open-ended responses into themes and
conducted a key driver analysis to understand which themes drive voting intention.
CASE STUDY : BENEFITS OF AI PROBING

10. 10
Proprietary & Confidential
Please watch video recording of this
session to see the demo of inca’s
Targeted and Trained probing

11. 11
Proprietary & Confidential
INCA SMARTPROBE AI PROBING – ADDITIONAL FEATURES
Quality Check
A ‘demerit’ feature determines quality of OE response based on detection of uninformative, gibberish or
irrelevant answers. Enabling live screening out of poor quality survey respondents
Trained Models
The researcher is able to provide examples of the types of probes they'd like to ask for certain respondent
answers, to train the AI to deliver even more useful probes for a particular topic:
• Specialised audiences
• Personality and Localisation
• Steering the probing – e.g. to present counter arguments
Targeted probing
The researcher is able to specify targets (words) of interest and trigger actions, e.g. Avoid, Probe on Detection,
Probe on Absence

12. 12
Proprietary & Confidential
Generative AI can help us
understand humans better, by
providing deep insight about them,
in their own words at a huge scale.
Whilst some applications of generative AI can be
seen as de-humanising; for market research,
generative AI creates the opposite opportunity.

13. 13
Proprietary & Confidential
phil@nexxt.in
Get deeper insight, faster
with
Powered by

14. Q & A
Ray Poynter
NewMR
Phil Sutcliffe
Nexxt Intelligence

15. Sponsors