What role can generative AI, such as ChatGPT, play in producing academic content that can be taught to students? This session explores the results of a mixed-methods study evaluating the comparative performance of human-generated and AI-generated educational materials. Through a mixture of psycholinguistic analysis of AI- and human-generated teaching content and a quantitative survey of their impact on students, we examine the capabilities and limitations of generative AI as a tool to deliver higher education.

1. derby.ac.uk
Sensitivity: Internal
A comparative sentiment analysis of human-
generated and machine-generated educational
content and their differential impact on students’
experience and learning.
Gary F. Fisher, Dean Fido, Paula Shaw

2. derby.ac.uk
Sensitivity: Internal
A comparative sentiment analysis of human-
generated and machine-generated educational
content and their differential impact on students’
experience and learning.
• Is it possible to engineer prompts into ChatGPT in such a way that it produces
online teaching content?
• What is the difference in how students respond to and judge human-written and AI-
written pieces of content?
• What aspects of academic content writing can Chat-GPT perform, and what
aspects can it not?

3. derby.ac.uk
Sensitivity: Internal
Methods
• Identify a small section of online content that is representative of the general
approach and style of the university's online content.
• Iteratively input prompts into ChatGPT-3 to ask it to produce an AI-
generated equivalent to that content.
• Conduct comparative linguistic analysis of human-written and AI-written texts using
LIWC-22.
• Conduct quantitative analysis of student judgements towards human-written and
AI-written texts using the survey tool prolific.
• Conduct qualitative analysis of students' reactions to and attitudes towards human-
written and AI-written texts using semi-structured interviews.

4. derby.ac.uk
Sensitivity: Internal
Human-generated production process

5. derby.ac.uk
Sensitivity: Internal
Prompts to ChatGPT
Prompt 1: Subject matter and length
“Write 700-750 words explaining the engineering and motivational approaches to
work design” (12:48, 03/03/2023).
Prompt 2: Audience
“Write 700-750 words of educational content that explains the engineering and
motivational approaches to work design to undergraduate students.” [12:58,
03/03/23]
Prompt 3: Context
“Write 700-750 words of educational content that explains the engineering and
motivational approaches to work design to undergraduate students studying an
online module in Business Psychology as part of an online Bachelors in
Business and Management.” [13:03 03/03/23]

6. derby.ac.uk
Sensitivity: Internal
Prompts to ChatGPT
Prompt 4: Format
“Write 700-750 words of educational content that explains the engineering and motivational
approaches to work design to undergraduate students studying an online module in
Business Psychology as part of an online Bachelors in Business and Management. The
text should use academic citations as appropriate and deploy real-world examples
of the engineering and motivational approaches in practice.” [13:06, 03/03/23)
Prompt 5: Educational features
“Write 700-750 words of educational content that explains the engineering and motivational
approaches to work design to undergraduate students studying an online module in
Business Psychology as part of an online Bachelors in Business and Management. The
text should use academic citations as appropriate and deploy real-world examples of the
engineering and motivational approaches in practice. After explaining each approach,
the text should include questions aimed at students that ask them to reflect on the
strengths of each approach and their own experience of it.” [13:20, 03/03/23]

7. derby.ac.uk
Sensitivity: Internal
LIWC-22 Results
Hypotheses:
- More language associated with sentiment and
narrative within the human-generated text.
Methods:
- Business Psychology text (human- and AI-generated).
- Subject texts to analysis by LIWC-22.
- Check for errors
Results:
• Similar usage of language associated with the labels
'Analytic' and 'Clout'.
• AI-generated markedly exceeds human-generated
text in use of language associated with the labels
'Authentic' and 'Tone'.

8. derby.ac.uk
Sensitivity: Internal
Quantitative Analysis
Hypotheses:
- More positive judgements towards human-generated
texts; especially when given a congruent label.
Methods:
- N = 361 UK-Based students via Prolific.
- Business Psychology text (human- and AI-generated).
- Participants saw 1 of 4 texts, provided judgements, and
answered questions on motivations for learning,
acceptance of AI, and demographics.
- Judgements were derived from NSS questions.
Results:
- Main effect of Generator F(1, 352) = 11.91, p = .001, ηp2 = .033
- No main effect of Label nor Interaction
- More favorable judgements (overall) associated with
intrinsic motivation and acceptance of AI
Generator
Average
Judgements

9. derby.ac.uk
Sensitivity: Internal
Findings and Recommendations
• With the correct prompts, ChatGPT can produce text that resembles the style and
approaches of online content in a short amount of time.
• This AI-generated content can use language associated with narrative and
sentiment in excess of human-generated content.
• Quantitative analysis suggests students hold more favourable judgements towards
AI-generated content than human-generated content.