Presented at GOR 23 on 21 September 2023. We present four uses cases for applying Generative AI, such as ChatGPT / GPT-4, to the quantitative analysis of survey data, including examples of prompts and generated output. This work was presented in the session on 'Innovation in Practice (2): Opportunities and Challenges' at the General Online Research Conference 2023 (https://www.conftool.org/gor23/index.php?page=browseSessions&form_session=77). Supporting write-ups are available at https://www.inspirient.com/case-studies/survey_analysis_automation.php and https://www.inspirient.com/case-studies/survey_quality_assurance.php -- Full abstract below -- Relevance & Research Question: Rigorous, well-documented data analysis is a corner stone of opinion research. ChatGPT and GPT-4 are the most recent additions to the toolset of every researcher/social scientist and are thought to have the potential to boost efficiencies by 50% and more – but they have trouble reliably adding more than three numbers, let alone apply complex statistical methods! This talk takes a use-case-driven approach to answering the following research questions: How can current Large Language Models (LLMs), like ChatGPT / GPT-4, be applied to rigorous analysis of survey data? Which use cases are feasible, and under which tradeoffs? Methods & Data: We've jointly identified four major use cases for LLMs applied to the analysis of raw survey data: 1) Natural-language data exploration, 2) desk research support, 3) generation of deliverables, and 4) LLM chatbots as client-facing deliverables. For each of these use cases, we've preprocessed samples of raw survey data in a fully automated process to allow current LLMs to be applied, as per the use case requirements. We've validated the textual results across teams to understand their respective potential and shortcomings, and consequently improved both our automated preprocessing as well as LLM prompting. Results: The results across the four use cases are impressive: Given nothing more than raw survey data (in form of an SPSS or Microsoft Excel file), we were able to reliably use LLMs to produce relevant output for each use case. The key aspect of our approach is the capability to link each LLM statement back to its original analytical results, thereby establishing a chain of trust into each statement of the LLM conversation that regular LLM use would lack otherwise. Added Value: At this time it's hard to fully estimate the automation potential&resulting efficiency gains for research agencies. We can however state with confidence, that the provided samples for all four use cases were generated within about one hour, for the given SPSS/Excel input data. In our talk, we'll demonstrate this process step-by-step, and together – based on a live demo – explore potential and limitations of this approach.

1. Trustworthy Analytics with Generative AI:
Four Use Cases for ChatGPT / GPT-4
Dr. Georg Wittenburg ▪ Inspirient
General Online Research Conference 2023 (GOR 23)
Kassel, 21 September 2023

2. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
General Online Research Conference 2023 (GOR 23), 21.09.2023 1
Dr. Georg Wittenburg
Co-founder and CEO at Inspirient

3. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
AI-supported Survey Analytics
Brief recap
Six Reasons
for AI / Automation
1) Improved Efficiency
Deliver detailed analytical evaluations in less
time and with more deep-dives
2) Quick Deliverables
Quickly deliver key results to clients, based on
pre-produced stories and prioritized insights
3) Preview Results during Fieldwork /
Data Collection
Preview results with client to gather feedback
and steer project
4) Advanced Data Validation
Collected data is more thoroughly checked for
data quality issues
5) Analytical Methods, Applied at Scale
Valuable insights surfaced with automated
advanced methods
6) Paper Trail for Every Result
Detailed, audit-proof documentation how each
analytical result was derived
General Online Research Conference 2023 (GOR 23), 21.09.2023 2
Survey raw data
(SPSS, Excel, CSV, …)
AI-supported
Survey Analytics
Client-facing deliverables
(PowerPoint, Excel, etc. in client format)

4. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
AI-supported Survey Analytics @ GOR
Four sessions at GOR 23 on 20-22 September
General Online Research Conference 2023 (GOR 23), 21.09.2023 3
Analysis
(qualitative / quantitative)
Synthesis
with Generative AI
Control
Hands-on
Experience
Community
Discussion
“Insights beyond Human Intuition”
(Talk at GOR 22)
“Trustworthy Analytics with Generative AI:
ChatGPT/GPT-4 Hands-on”
(Workshop, 20 Sep 2023, 13:30, Room 1124)
Panel Discussion on AI / LLM
(22 Sep 2023, 11:45, Room 1101)
“Standardized Annotations for Survey Datasets:
Enabling Automated Quality Assurance and Evaluation”
(Talk, 21 Sep 2023, 12:00, Room 1124)
“Trustworthy Analytics with Generative AI:
Four Use Cases for ChatGPT/GPT-4”
(Talk, 21 Sep 2023, 17:00, Rooms 1117/1118)
current session

5. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
Why are we doing this?
Our Existing AI-derived Results Already Had Substantial Text
General Online Research Conference 2023 (GOR 23), 21.09.2023 4
Key insight in
generated slide title
Brief method description
in generated chart title
Data processing steps
and analytical in
generated footnote
Description of key
patterns as part of
generated chart
annotations
Example: Generated slide for Berkeley IGS Poll Q4 2021
Detailed notes in
generated tabular
exports
So how hard could it be to generate input for ChatGPT?

6. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
Approach
AI-derived Results Are Fed into LLM, e.g., ChatGPT, for Discussion
General Online Research Conference 2023 (GOR 23), 21.09.2023 5
Survey
raw data
Analytical results
inspirient
Secure Large
Language Model1
(e.g., dedicated
ChatGPT instance)
User
1. Running as a dedicated isolated instance, i.e., without leaking internal data
System Architecture
This technical setup ensures that no data or chat
messages are leaked beyond the dedicated model.

7. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
Example
Conversation with ChatGPT about Automatically Generated Results
General Online Research Conference 2023 (GOR 23), 21.09.2023 6
Citation markers link
LLM output to our
validated results and
allow us to include
pre-generated visuali-
zations

8. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
State of the Art (1/2)
Generative AI: Your Mileage May Vary
General Online Research Conference 2023 (GOR 23), 21.09.2023 7
Users get tricked by appearance. Our user
study results show that users prefer ChatGPT
answers 34.82% of the time. However, 77.27% of
these preferences are incorrect answers. […]
During our study, we observed that only when the
error in the ChatGPT answer is obvious, users
can identify the error.
v.d. Heide, Wenz, Haensch (LMU)
GOR ‘23 (downstairs)
Who Answers It Better? An In-Depth Analysis of ChatGPT
and Stack Overflow Answers to Software Engineering Questions
Kabir, Udo-Imeh, Kou, Zhang (Purdue)
https://doi.org/10.48550/arXiv.2308.02312

9. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
State of the Art (2/2)
Generative AI: Know Your Use Case
General Online Research Conference 2023 (GOR 23), 21.09.2023 8
The trick to making the most of AI is to approach
it as what it was intended to always be, a tool,
not a complete solution. Working out how best to
integrate that tool into the research process
makes it, in the words of Richard Colwell of
RedC, “a game-changer”.
The Boston Consulting Group, 21 September 2023
https://www.bcg.com/publications/2023/
how-people-create-and-destroy-value-with-gen-ai
The human effect? Putting AI to the storytelling test
Lucy Davidson (Keen As Mustard Marketing), ESOMAR 2023
https://researchworld.com/innovations/
the-human-effect-putting-ai-to-the-storytelling-test

10. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
Generative AI / ChatGPT: Four Use Cases
General Online Research Conference 2023 (GOR 23), 21.09.2023 9
1 2 3 4
Typical
Requests
Value
Generated
Please list the Top 5 findings
in this survey!
How would the potential
campaign messages based
on this survey align with
Congressional Republicans?
Which channels may a
Republican candidate chose
to most effectively communi-
cate these messages?
Based on the overall findings
in this survey, what would
good outline items a 4-section
long-form report?
Could you please write a short
two-paragraph draft outline for
item #2?
I'm Gavin Newsom, Governor
of California. What should I
know about this survey?
That sounds all very technical
to me! Can you give me three
bullets on how I could win this
recall election?
Which findings in this survey
are related to age groups?
Which findings tell me about
plans to vote in the recall
election?
• Quick exploration of survey
results with a natural-
language interface
• More dynamic and more
contextually aware than any
dashboard
• Contextualization of survey
results by applying external
general knowledge
• Hypothesis generation for
recommendations based on
survey results
• Automated generation of
draft deliverables, e.g., …
 Executive summaries,
 Long-form reports
 Press releases / articles
…incl. result visualizations
• New type of deliverable, far
beyond tabulation, reports of
dashboards, etc.
• Enables clients / decision
makers to directly interact
with the survey results
Natural-language
Exploration of Results
Desk Research /
Results in Context
Analysis-driven
Deliverables
Per-Survey Chatbots
as Deliverables

11. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
General Online Research Conference 2023 (GOR 23), 21.09.2023 10

12. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
11
1 Natural-language
Exploration of Results
Full transcript
available online
General Online Research Conference 2023 (GOR 23), 21.09.2023

13. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
12
2 Desk Research /
Results in Context
Full transcript
available online
General Online Research Conference 2023 (GOR 23), 21.09.2023

14. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
13
3 Analysis-driven
Deliverables
Full transcript
available online
incl. sample draft
deliverables for...
• Executive summary
• Report outline
• Report section
• Translation of section
• Press release
• Social media posts
• Email to client
General Online Research Conference 2023 (GOR 23), 21.09.2023

15. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
14
4 Per-Survey Chatbots
as Deliverables
Full transcript
available online
Audio sample
available online

16. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
The Road Ahead
AI-supported Analyses across Multiple Surveys
General Online Research Conference 2023 (GOR 23), 21.09.2023 15

17. Copyright
©
2023
by
Inspirient
GmbH.
All
rights
reserved.
Thank you for your time today
General Online Research Conference 2023 (GOR 23), 21.09.2023 16
Dr. Georg Wittenburg
georg.wittenburg@inspirient.com