Bing Chat analyzed an argumentative article by generating an argument map in ArgDown syntax highlighting the key claim, premises, and their relationships. However, it also introduced some fallacious elements like incorrect summaries, commentary nodes, and incorrect argument classifications. While this shows generative AI has potential for argument analysis, more evaluation is needed to address inaccuracies that could undermine critical thinking. With improvements, it may help facilitate deliberations by preparing resources or assisting engaged citizens in analyzing complex arguments.

1. UTS CRICOS 00099F
Could Generative AI Augment
Reflection, Deliberation and
Argumentation?
Simon Buckingham Shum
Professor of Learning Informatics &
Director, Connected Intelligence Centre
https://cic.uts.edu.au
15th May 2023
Critical Deliberative Democracy Tech Workshop • 11th Int. Conf. Communities & Technologies

2. How to support the analysis of these arguments?
https://futureoflife.org/open-letter/pause-giant-ai-experiments/ https://www.dair-institute.org/blog/letter-statement-March2023
challenges

3. Generate the ArgDown syntax description for the key arguments

4. Paste the GPT-generated code into the ArgDown web app à Argument Map

5. GPT-generated Argument Map
(green) Elements classified
by Argumentation Scheme
article à GPT analysis à code à visualization
Claim in the original letter
which is not contested
(green supporting premises)
(white) Claims
and Premises

6. Questioning GPT on why it added (unrequested)
Argumentation Scheme classifications to (green) nodes

7. GPT inserts the Critical
Questions in the Argument Tree
(an alternative view in ArgDown to the graph)

8. Additional step:
Hide the Critical Questions as
an example student activity

9. Evaluating the Argument Map
X
i
X
X
Correct summary of authors
Hallucination
https://www.reasoninglab.com/patterns-of-argument/argumentation-schemes/waltons-argumentation-schemes/
Fallacy of Fallacy of omission
X
Incorrect term
i
Ad hominem
i “commentary” from Bing

10. When challenged about a node in the Argument Map, Bing Chat
fabricates a quote from the article, and refuses to back down!

11. Conclusions (1/2)
Bing Chat (a version of GPT-4) showed the capability to analyse an argumentative article:
• extracted the key claim and underlying premises, summarising them in own words
• generated markdown (ArgDown) showing supporting/challenging relationships
• (without being asked to) attempted to classify some nodes using Walton’s Argumentation Schemes.
Bing Chat also introduced:
• fallacious nodes (incorrect summaries of the authors, and incorrect commentary nodes), links, and
argument classifications (inventing argument types, and/or misclassifying nodes)
This is an exploratory example, and more systematic Argument Mining evaluations are required

12. Conclusions (2/2)
We’ve argued that embracing imperfection* in tech can be productive if it promotes deeper critical
thinking in learners, e.g., learning by correcting the automated output, or reflecting on questions it asks,
or why it seems wrong. Students can be directed and scaffolded to engage in such activity.
This is permissible in formal education, but may be less attractive to citizens engaged in a policy
deliberation, many of whom lack the internal or external motivation to think that hard.
But assuming future tools give more accurate argument maps/outlines, perhaps we can see use-cases
including:
• assisting facilitators/educators to prepare learning resources for civic deliberations
• assisting very engaged citizens to dissect complex arguments, and perhaps lowering the entry
threshold for others who might otherwise not engage with such structured, critical deliberation
• an article is very different to a multi-author conversation, but we can envisage summarising online
discussions (NB: Teams is starting to summarise topics and actions in meeting transcripts)
For this analysis see https://simon.buckinghamshum.net/2023/05/conversational-genai-for-argument-analysis
* Kitto, K., Buckingham Shum, S., & Gibson, A. (2018). Embracing imperfection in learning analytics. Proceedings 8th International
Conference on Learning Analytics and Knowledge (LAK18), Sydney, AUS. ACM, NY, pp. 451–460. https://doi.org/10.1145/3170358.3170413