«Conversational agents in MOOCs: what’s the point?» / Stavros Demetriadis, associate professor at the School of Informatics, Aristotle University of Thessaloniki (AUTh), Greece.

ARISTOTLE UNIVERSITY OF THESSALONIKI
Conversational agents in MOOCs:
What’s the point?
Stavros Demetriadis
Assoc. Professor
School of Informatics
Aristotle University of Thessaloniki
Greece

ARISTOTLE UNIVERSITY OF THESSALONIKIStavros Demetriadis @ eMadrid, May 11, 2018
Aristotle University
of Thessaloniki (AUTh)
 Est. 1926
 42 Schools in 11 Faculties
 Academic Staff: 2024
 Students: ~ 75000

Software and Interactive Technologies
Lab (SWITCH)
SWITCH Lab
 Software Technology
 Learning Technologies
 Music Informatics
 Est. 1992
 Academic Staff: 30
 Students: ~ 900

Overview
 Peer interaction: productive dialogue for learning
 Conversational Agents
 MentorChat: Research Evidence
 colMOOC project: Agents in MOOCs
 Design, Expectations, Research
 Erasmus+
 Knowledge Alliances

Peer interaction
 Peer interaction is the key learning mechanism
for knowledge building in collaborative learning
settings
 The real generative processes of the emergence
of mind and the production of knowledge can
be usefully modeled as multilevel conversations
between conversants…
 Conversation Theory (Boyd, 2001)

Free = suboptimal collaboration
 But no guarantee that these interactions will actually occur
 Dillenbourg & Tchounikine (2007)
 Various studies have identified patterns of suboptimal
collaboration in free (non-supported) collaboration
conditions
 (e.g., Liu & Tsai, 2008).

Scripting collaborative learning
 Scaffolds are needed to increase the probability that
productive peer interactions occur…
 … such as consensus building, explicit explanation, mutual
regulation, argumentation, conflict resolution, etc…
 For example ‘Make it Explicit!’: …when asking students to
proactively articulate their own positions explicitly, then
improved peer interaction is triggered in a subsequent
collaborative session
 Papadopoulos, Demetriadis & Weinberger, 2013

When students discuss…
Teachers may intervene…
 This kind of intervention fosters productive dialogue and
deeper learning: triggers students to elaborate, recall, make
connections, argue, etc.
 Do you agree…?
 Do you think this
relates also to…?
 Do you think this
also relates to… ?
 ………………..

“Academically Productive Talk” (APT)
 Framework originating from the teachers’ community
emphasizing the orchestration of classroom discussions
 (Michaels et al. 2010).
 Addresses a set of teachers’ discussion practices (‘moves’)
that can lead to participation of all learning partners.
 Prioritizes reasoning over correctness
 Highlights ‘sharing their reasoning out loud’ as an effective
teaching strategy for the elicitation of learners’ perspectives
 (Papadopoulos, Demetriadis, & Weinberger, 2013).

APT Move Example
Link Contributions
Agree-Disagree
Add-On
A. “Do you agree with what your partner said…?”
B. “Would you like to add something to …?”
Re-voice “So, are you saying that … Is that correct?”
Ask for Accuracy
Ask for Credibility
Ask for
Completeness
A. “Could you identify that in a reference book?”
B. “This is probably true, but how could we get
more evidence on that?”
Build on Prior
Knowledge
“How does this connect with what we know about
…?”
Ask for Reasoning
“What are the arguments in favor of that?”, “Why
do you think that?”
Expand Reasoning
Take your Time
Say More
A. “Please take your time before answering”
B. “That's interesting! Can you elaborate on that?”
APT Moves

APT move efficacy
 The efficacy of each APT move depends on
various factors like:
 the teacher authority
 the student background
 the education level
 the difficulty of the instructional domain
 the students’ knowledge background.
 Michaels et al. (2008)

When you have too many students…
 MOOCs are
popular today
 Open
Educational
Resources
 How can we
advance peer
interaction and
productive
dialogue in
MOOCs?  Can the Teacher make APT interventions
in MOOC chats? No! :-(

…but someone else can do it for you…
a conversational agent

Agents are friendly creatures that live
onscreen…
 A ‘pedagogical agent’ is considered
an autonomous computer-generated
virtual character aiming to fulfill
specific pedagogical purposes in a
learning environment
 Gulz, Haake, Silvervarg, Sjödén, & Veletsianos (2011).

Conversational Agents (CAs):
Agents that talk to learners
 Agents attempting to
engage learners in a
conversation through
natural language

Can CAs successfully support
peer interaction? Yes!
 Agent directed learners through prescribed lines
of reasoning  substantially improved learning
 Chaudhuri (2009)
 Agent displayed reflective prompts while learners
took turns tutoring each other  increased the
conceptual content in students’ utterances
 Walker et al. (2011)
 Agent interacted with peers during collaborative
brainstorming  significantly enhanced learners’
creativity (more ideas as compared to
brainstorming with a human peer)
 Wang et al. (2007)

Emerging research direction:
 Agile form of conversational agent to support social
interaction by inducing constructive mental processes
 Conversational agent to scaffold collaborative learning
discussions through the approach of ‘Academically
Productive Talk’ (APT)

MentorChat
A configurable conversational agent
 APT-based: Makes APT interventions during synchronous online
student discussion (chatting)
 Configurable: (1) various APT moves (2) teacher enters the domain

Configurable-1: can be setup to make
various moves
Build on Prior Knowledge
Link contributions
1. Agree/Disagree
2. Add on

Configurable-2: The teacher enters the
domain

MentorChat: Key design decisions and
models
 Moderate AI level
 Agent does not enter into
full fledged discussion
 Models and triggers
student dialogue by
making APT moves
 Development efficiency
 Teacher configures
the domain
 Agent as a teacher-
configured learning tool
 NOT teacher substitute

Study 1: Linking Contributions
(Tegos, Demetriadis & Karakostas, 2015)
 Participants: 43 undergraduate CS students
 Randomly assigned to groups and 2 conditions:
 Treatment (received agent LC interventions, 9 dyads and 1 triad)
 Control condition (no Agent, 11 dyads)

Example: Agent LC intervention
User Student Dialogue with Agent Interventions
George What is the most relevant design principle here?
Anna
That preferably the information should be displayed using different
representation codes, such as images, diagrams, etc., in order to achieve a
better understanding. I think this is the principle of spatial contiguity.
George I see
Agent
George, do you agree or disagree with Anna’s statement regarding the
spatial contiguity principle? Please, elaborate.
George
[Submitted Response to Agent] Well, I think the principle of spatial
contiguity suggests that the text and the relevant image should be put
close to each other
Anna
Ah yes! That’s true. I believe this design principle should be our
number one priority because Mayer has shown that students tend to learn
better when text and graphics are placed close to one another than when
they are placed far from each other (avoid screen scrolling).

Study 1: Key Results
 Learning: Treatment outperformed control in both individual and
group learning measures (conceptual knowledge)
 Explicitness: Treatment significantly increased frequencies of
explicitness (explicit positions and explicit arguments during
discussions)
 Agent intervention triggered on average 1.18 subsequent explicit
contribution (Explicit Response Ratio, ERR = 1.18)
 Treatment students used considerably more key domain concepts in
their group discussions
 Frequency of explicit arguments was identified as mediator variable

Explicitness as mediator variable
 (Tegos, Demetriadis & Karakostas, 2015)

Study 2: Building on Prior Knowledge
(Tegos & Demetriadis, 2015)
 Participants: 72 undergraduate CS
students
 Randomly assigned to groups and 2
conditions:
 Treatment (received agent BPK interventions,
19 dyads)
 Control condition (no Agent, 17 dyads)
 Confirmed all study-1 results
 Improved Individual and Group learning
 Increased explicitness
 Frequency of explicit arguments was identified
as mediator variable

Study 3: Solicited (A) vs. Unsolicited (B)
Interventions (Tegos, Demetriadis, & Karakostas, 2014)
Unsolicited
interventions:
 Perceived by the
students as more
intrusive
 But found to be
more effective
in stimulating
explicit
reasoning
A B

Study 4: Undirected (A) vs. ‘Weak’ Directed (B)
Interventions (Tegos, Demetriadis, & Tsiatsos, 2014)
 ‘Weak’ Directed interventions increase the transactive quality
of a peer dialogue and, leads to improved learning
 Transactivity: Reasoning on each others contributions

Lessons from MentorChat
 Conversational Agent interventions based on APT can
improve learning outcomes in online student discussions
 Interventions trigger students’ explicit reasoning  results in
improved learning
 Unsolicited interventions >> Solicited
 ‘Weak’ Directed interventions >> Undirected

Limitations
 Controlled conditions: Participants were informed that
discussions would be reviewed by the course instructor…
 .. learning benefits though were observed without any feedback from
the instructor
 Moderate Agent AI: Not full-fledged discussion with students
but only trigger peer interaction
 Objective: easily constructed and deployed agents with a minimum
required level of AI
 Text-based communication (use of keyboard):
 Easier for tech-savvy students
 Can be extended to voice-based dialogue?

Portrait of a Scholar
by Domenico Feti, Italian painter
Gemäldegalerie, Dresden
What
next?
 The colMOOC project to integrate similar
agent-based tools in MOOCs
(Demetriadis et al. 2018)

The point is….
We expect….
 Learner engagement and satisfaction through social interaction
 Learning benefits at cognitive (domain) and metacognitive level
 Developing a community of educators to further explore the
impact of agents on learning
 To engage MOOCs learners in
academically productive
dialogue triggered by agent
interventions

colMOOCs to test-bed the approach
 Develop pilot MOOCs in
3 domains
 With agent-based chat
integrated as learning
activity

Key design features:
1) Teacher configures the agent domain
 The teacher as a creator-developer, the animator of the
agent

2) The agent as an interaction mediator
 The agent operates as a TIM (“Teacher-configurable
Interaction Mediator”) mediating teacher-group interactions

3) Analytics component
 Empowered also by learning analytics tools to visualize
peer-agent interaction data

4) Community building
 Agent-based chat software freely available also independent
of a MOOC platform
 Developing their own agent…
 Upload it to an agent service/repository…
 Use agent they find useful…
 Extend agent domain models or change ‘behavior’ (APT
moves)
 In the future: Integrate additional cognitive or metacognitive
capabilities in the agent

 Educational Design  Software development, System
integration, Pilot MOOCs development (M1-M18, June 2019)
 Pilot MOOCs Deployment and Evaluation  Community
building  Dissemination (M19-M36, December 2020 and beyond..)

Open design/research issues
 Teacher tools: How to setup the agent and model the domain?
 Group formation: How to match peers? Group size?
 Learners’ feedback: How to provide it? Peer assessment?
 Gamification? Increase engagement?
 Modalities of peer-agent communication? (verbal)
 Support all forms of knowledge? (conceptual, procedural)
 Synchronous vs. Asynchronous peer discussions?

Thank you! Questions?
sdemetri@csd.auth.gr
http://mlab.csd.auth.gr/sdemetri/

References 1/2
 Boyd, G. M. (2001). Reflections on the conversation theory of Gordon Pask. In R. Glanville & B. Scott (Eds.), Festschrift in
celebration of Gordon Pask. Kybernetes: The International Journal of Systems and Cybernetics 30(5–6), 560–570.
 Chaudhuri, S., Kumar, R., Howley, I., & Rosé, C. P. (2009). Engaging collaborative learners with helping agents. In V.
Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial
Intelligence in Education (pp. 365–372). Amsterdam: Ios Press.
 Demetriadis, S., Karakostas, A., Tsiatsos, Th., Caballé, S., Dimitriadis, Y., Weinberger, A., Papadopoulos, P.M.,
Palaigeorgiou, G., Tsimpanis, C., & Hodges, M., (2018). Towards Integrating Conversational Agents and Learning
Analytics in MOOCs. In L. Barolli et al. (Eds.): Proceedings of EIDWT 2018, Springer, LNDECT 17, pp. 1–12.
https://doi.org/10.1007/978-3-319-75928-9_98
 Dillenbourg, P., & Tchounikine, P. (2007). Flexibility in macro-scripts for computer-supported collaborative learning.
Journal of computer assisted learning, 23(1), 1–13.
 Fischer, F., Kollar, I., Stegmann, K., & Wecker, C. (2013). Toward a script theory of guidance in computer–supported
collaborative learning. Educational Psychologist, 48(1), 56–66.
 Gulz, A., Haake, M., Silvervarg, A., Sjödén, B., & Veletsianos, G. (2011). Building a social conversational pedagogical
agent–design challenges and methodological approaches. In D. Perez–Marin & I. Pascual–Nieto (Eds.), Conversational
agents and natural language interaction: Techniques and effective practices (pp. 128–155). Hershey, PA: IGI Global.
 Liu, C. C., & Tsai, C. C. (2008). An analysis of peer interaction patterns as discoursed by on–line small group problem–
solving activity. Computers & Education, 50(3), 627–639.
 Michaels, S., O’Connor,M. C., Hall, M.W., & Resnick, L. B. (2010). Accountable talk sourcebook: For classroom that works
(v.3.1). University of Pittsburgh Institute for Learning. Retrieved March 1, 2015, from
http://ifl.pitt.edu/index.php/download/index/ats

References 2/2
 Michaels, S., O’Connor, C., & Resnick, L. B. (2008). Deliberative discourse idealized and realized: Accountable talk in the
classroom and in civic life. Studies in Philosophy and Education, 27(4), 283–297.
 Papadopoulos, P. M., Demetriadis, S. N., & Weinberger, A. (2013). ‘Make it explicit!’: Improving collaboration through
increase of script coercion. Journal of Computer Assisted Learning, 29(4), 383–398.
 Tegos, S., Demetriadis, S. N., & Karakostas, A. (2014). Conversational agent to promote students’ productive talk: The
effect of solicited vs. unsolicited agent intervention. In IEEE 14th International Conference on Advanced Learning
Technologies (pp. 72–76). Athens, Greece.
 Tegos, S., & Demetriadis, S. (2015). Promoting academically productive talk with conversational agent interventions in
collaborative learning settings. Computers & Education, 87, 309–325.
 Tegos, S., Demetriadis, S., & Tsiatsos, T. (2014). A configurable conversational agent to trigger students’ productive
dialogue: A Pilot Study in the CALL domain. International Journal of Artificial Intelligence in Education, 24(1), 62–91.
 Walker, E., Rummel, N., & Koedinger, K. R. (2011). Designing automated adaptive support to improve student helping
behaviors in a peer tutoring activity. International Journal of Computer-Supported Collaborative Learning, 6(2), 279–306.
 Wang, H. C., Rosé, C. P., Cui, Y., Chang, C. Y., Huang, C. C., & Li, T. Y. (2007). Thinking hard together: The long and
short of collaborative idea generation in scientific inquiry. In Proceedings of the 8th International Conference on Computer–
Supported Collaborative Learning (pp. 754–763). International Society of the Learning Sciences. New Brunswick, NJ.

«Conversational agents in MOOCs: what’s the point?» / Stavros Demetriadis, associate professor at the School of Informatics, Aristotle University of Thessaloniki (AUTh), Greece.

Recommended

Recommended

More Related Content

Similar to «Conversational agents in MOOCs: what’s the point?» / Stavros Demetriadis, associate professor at the School of Informatics, Aristotle University of Thessaloniki (AUTh), Greece.

Similar to «Conversational agents in MOOCs: what’s the point?» / Stavros Demetriadis, associate professor at the School of Informatics, Aristotle University of Thessaloniki (AUTh), Greece. (20)

More from eMadrid network

More from eMadrid network (20)

Recently uploaded

Recently uploaded (20)

«Conversational agents in MOOCs: what’s the point?» / Stavros Demetriadis, associate professor at the School of Informatics, Aristotle University of Thessaloniki (AUTh), Greece.