Keynote presentation from the Collaborative Learning Analytics Workshop at CSCL 2019 in Lyon, France.

1. Towards Collaborative
Learning Analytics
Opportunities, Challenges and
Tensions at the Intersection of
CSCL and LA
Alyssa Friend Wise
Director, NYU Learning Analytics Research Network
Associate Professor of Learning Sciences & Educational Technology
New York University

2. An Inflection Point for CSCL
25+ years of work on how people learn together
Computer-Supported Collaborative Learning, digitally mediated team learning,
collaborative problem solving, computer-supported collaborative work,
collaboration in learning sciences, learning at scale, problem-based learning
educational technology, online teaching and learning
Rapid technological innovation and societal infusion
social networking, data-enabled mobile communication, interactive tabletops,
sensor technologies, virtual, augmented and mixed reality, data stream capture,
machine learning, analytics, artificial intelligence
Multiplicity of goals, processes & constructs of interest
joint attention, cognitive presence, argumentative knowledge construction,
content expert, promisingness, accountable talk, improvable ideas, uptake,
epistemic frame, transactivity

3. Building the Future (Visions of CSCL)
Thrust 1: Engage the Evolving Ecosystem
Create a shared taxonomy of CSCL support (#1, SQ-R)
Scrupulously scrutinize “collaboration” and “community” (#3)
Consider reconfigurable constellations of collaborators (#7)
Support access and equity for underserved populations (#8)
Wise, A. F. & Schwarz, B. S. (2017). Visions of CSCL: Eight Provocations for the Future of the Field.
International Journal of Computer-Supported Collaborative Learning, 12(4), 423-467.
Rummel, N. (2018). One framework to rule them all? Carrying forward the conversation started by Wise
and Schwarz. International Journal of Computer-Supported Collaborative Learning, 13(1), 123-129.

4. Thrust 2: Analytics, Adaptivity & Agency
Vigorously pursue computational approaches (#5)
Integrate analytical and interpretive methods (#4)
Make analytic feedback and adaptive support a top priority (#6)
Prioritize individual and shared learner agency (#2, SQ-T)
Wise, A. F. & Schwarz, B. S. (2017). Visions of CSCL: Eight Provocations for the Future of the Field.
International Journal of Computer-Supported Collaborative Learning, 12(4), 423-467.
Tchounikine, P. (2019). Learners’ agency and CSCL technologies: towards an emancipatory
perspective. International Journal of Computer-Supported Collaborative Learning, 1-14.
Building the Future (Visions of CSCL)

5. Should CSCL
Embrace Learning Analytics?
How
Learning analytics offer powerful methods for
identifying patterns in large amounts of data and
leveraging them to inform in-progress learning
But are these methods appropriate and useful for
generating deep insight into complex processes of
collaborative learning and supporting thoughtful
self- co- and shared regulation?
Wise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.

6. Three Key Concerns
Algorithmic Processing over Human Insight
– Leveraging both complementarily in DIPTiC method
– Extensive manual follow-up of computational results
Generalized Structures over Contextualized Processes
– Going back to the data to understand how top linguistic
model features were used by learners
– Following up on communities identified by SNA methods
to probe interactional processes
Empirical Findings over Theory Building
– Considering the meaning of different tie definitions
– Recognizing the need to reconceptualize learning in
MOOCsWise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.

7. Three Key Concerns
Algorithmic Processing AND Human Insight
– Leveraging both complementarily in DIPTiC method
– Extensive manual follow-up of computational results
Generalized Structures AND Contextualized Processes
– Going back to the data to understand how top linguistic
model features were used by learners
– Following up on communities identified by SNA methods
to probe interactional processes
Empirical Findings AND Theory Building
– Considering the meaning of different tie definitions
– Recognizing the need to reconceptualize learning in
MOOCsWise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.

8. Algorithmic Processing
AND Human Insight
• Apply sophisticated algorithms
to find patterns in large data
• Make decisions about comp.
methods, algorithm(s),
features, hyperparameters
• Interpret results in light of
existing knowledge base
Can also extend human coding
to scale and use humans to
verify / correct machine codes
(!) But high-level methodological
decisions play different role in
knowledge-generation than
researchers as instrument
Open learner models allow
students to inspect the
model and edit / negotiate it
Wise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.

9. Quantitative Methods EDS/LA Qualitative Methods
Focus on treatments and
outcomes
Focus on process of
learning
Identifying regularities
and patterns
Focus on nuances of
specific contexts
Generalized insights Particularized insights
Confirmatory Exploratory
Pre-determined analysis Emergent analysis
Generalized Structures
AND Contextualized Processes
Wise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.

10. • Importance of and need for increased attention to
theory acknowledged among (most) LA researchers
• New analytic methods can
spark theorization (e.g.
temporality)
• Computational models are powerful tool to instantiate
and examine theoretical models
Empirical Findings
AND Theoretical Contributions
Wise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.

11. Wise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.
Addressing the Concerns

12. Addressing the Concerns
Algorithmic Processing AND Human Insight
– Algorithmic triangulation with human reconciliation (DIPTiC)
– Manual follow-up of computational results
Generalized Structures AND Contextualized Processes
– Going back to the data to understand how top linguistic
model features were used by learners
– Following up on communities identified by SNA methods to
probe interactional processes
Empirical Findings AND Theoretical Contributions
– Considering the meaning of different tie definitions
– Recognizing the need to reconceptualize learning in MOOCs
Wise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.

13. Principles for Learning Analytics in CSCL
1. Ground analysis in theory
2. Characterize the context richly
3. Justify choice of data and/or features
4. Make sense of high-level patterns using low-level data
5. Present analytical results connected to learning processes
6. Appraise scope / boundaries of applicability
7. Consider theoretical implications
Wise, A. F. & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. ICLS 2018 (pp.1799-1806). London, UK: ISLS.

14. Collaborative
Learning Analytics
1. From constructs to clicks (and back again)
Analytics of Collaborative Learning
2. Making analytics actionable (really)
Collaborative Learning Analytics
Forthcoming chapter in the International Handbook of CSCL
coauthored with Simon Knight and Simon Buckingham Shum
Wise, A. F., Knight, S. & Shum, S. B. (forthcoming). Collaborative Learning Analytics.
International Handbook of Computer-Supported Collaborative Learning. Springer.

15. 1.
From constructs to
clicks (and back again)
Analytics of Collaborative Learning

16. Wise, A. F., Knight, S. & Shum, S. B. (forthcoming). Collaborative Learning Analytics.
International Handbook of Computer-Supported Collaborative Learning. Springer.
Derived FeaturesMetricsConstruct Digitally Captured Events
Joint Attention
Mechanism by
which a shared
reference helps
collaborators
coordinate with one
another to ground
communication
(Clark & Brennan, 1991;
(Tomasello, 1995)
Joint Visual
Attention
Shared visual
focus on a spatial
area can can act
as a proxy for
shared cognitive
attention.
Gaze Similarity /
Cross-Recurrence
Measure of overlap
in people’ fixations
on similar regions of
the screen within +/-
2s
(Schneider & Pea, 2013;
Sharma et al., 2015).
Fixations
Eye focus on a
specific location for
some period of time
Saccades
Movement that
repositions eye focus
to a new location
Analytics of…
g g g i

17. Wise, A. F., Knight, S. & Shum, S. B. (forthcoming). Collaborative Learning Analytics.
International Handbook of Computer-Supported Collaborative Learning. Springer.
Derived FeaturesMetricsConstruct
Leading Learner
The person who
initiates joint visual
attention (higher
learning gains in
pairs where this
role shared equally)
(Schneider et al., 2016)
Joint Visual
Attention Initiator
When overlap in
gaze occurs, the
person whose
gaze focuses on
the region first.
Gaze Similarity /
Cross-Recurrence
Measure of overlap
in people’ fixations
on similar regions of
the screen within +/-
2s
(Schneider & Pea, 2013;
Sharma et al., 2015).
Fixations
Eye focus on a
specific location for
some period of time
Saccades
Movement that
repositions eye focus
to a new location
Analytics of…
Digitally Captured Events
ig i g i

18. Wise, A. F. & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of
Learning Analytics (Special Section on Learning Analytics and Learning Theory), 2(2), 5-13.
Derived FeaturesMetricsConstruct
Cognitive Presence
Four-phase cycle
of critical thinking in
the CoI model involving
triggering, exploration,
integration, and
resolution
(Garrison, Anderson &
Archer, 2001)
Comparative
Word Type
Prevalence
Statistical measures
of relative word use
in particular phases
of cognitive presence
cycle
(Joksimovic, Gasevic,
Kovanovic, Adesope and
Hatala (2014)
Causal Words
because, hence
Exclusive Words
Without, but, exclude
Discrepancy Words
should, would, could
(Tausczik, & Pennebaker,
2010).
Forum Postings
Text of what was
said by whom when
Analytics of…
Digitally Captured Events
and in what order
g i i iig

19. Knowledge
Construction
Phase
Post Number

20. 1
2
3
4
5
0 5 10 15 20
1a
1
2
3
4
5
0 5 10 15 20
1b
1
2
3
4
5
0 5 10 15 20
2a
1
2
3
4
5
0 5 10 15 20
2b
1
2
3
4
5
0 5 10 15 20
3
1
2
3
4
5
0 5 10 15 20
4

21. 1
2
3
4
5
0 5 10 15 20
Sharing
Information
Negotiating
Meaning
Testing &
Modifying
Exploring
Dissonance
Agreeing &
Applying
Wise, A. F. & Chiu, M. M. (2011). Analyzing temporal patterns of knowledge construction in a role-based
online discussion. International Journal of Computer-Supported Collaborative Learning. 6(3), 445-470.
Level of Knowledge Construction Contribution by Post

22. 1
2
3
4
5
0 5 10 15 20
1a
1
2
3
4
5
0 5 10 15 20
1b
1
2
3
4
5
0 5 10 15 20
2a
1
2
3
4
5
0 5 10 15 20
2b
1
2
3
4
5
0 5 10 15 20
3
1
2
3
4
5
0 5 10 15 20
4
No Regressive
Segments
Pivotal Posts →
Distinct
Segments
No Regressive
Segments
Segments
Skipped
KC phases

23. Level of Knowledge Construction Contribution by Post
Wise, A. F. & Chiu, M. M. (2011). Analyzing temporal patterns of knowledge construction in a role-based
online discussion. International Journal of Computer-Supported Collaborative Learning. 6(3), 445-470.
1
2
3
4
5
0 5 10 15 20
Sharing
Information
Negotiating
Meaning
Testing &
Modifying
Exploring
Dissonance
Agreeing &
Applying

24. Ochoa, X et al. (2013). Expertise estimation based on simple multimodal features. Proceedings
of the 15th ACM on International Conference on Multimodal Interaction, 583-590
Derived FeaturesMetricsConstruct
Math “Expert”
In each problem
solving group, there is
one learner who the
others will defer to in
problem solving.
Calculating Time
Activity Level
Working Time
Number
Speech Duration
Numerals Mentioned
Writing Speed
Path Length
Calc Position + Angle
Difference Frame Sum
Head-Center Distance
Speech Units
Words Used
Stroke Unit
Stroke Coordinates
Video Capture
Audio Transcript
Digital Pen Trace
Analytics of…
Digitally Captured Events

25. MOOC Statistics Discussion
Content-Related Network Non-Content Network
Content-related network included
fewer learners but with higher
degree and edge weights
Wise, A. F., & Cui, Y. (2018). Learning communities in the crowd: Characteristics of content related
interactions and social relationships in MOOC discussion forums. Computers & Education, 122, 221-242.
What Data to Create Analytics Of?

26. MOOC Statistics Discussion
Content-Related Network Non-Content Network
Content-related network included
fewer learners but with higher
degree and edge weights
Content interactions had longer
threads with more repeat
participants, more complex topics
and greater social presence cues
Wise, A. F., & Cui, Y. (2018). Learning communities in the crowd: Characteristics of content related
interactions and social relationships in MOOC discussion forums. Computers & Education, 122, 221-242.
Can anybody help me with question 10 of unit 4? Do we
have to consider the mean = proportion = 112/200 = 0.56?
Good morning! The question states that you should use
the normal approximation to the binomial….the mean is
not a proportion, it is = n* p!
Thanks, but I'm still confused. Don't we have to use the
statistics of proportion here? 112/200 = 0.56 and if I'm
using the formula mean = n*p, and x = 112, then the z
score is coming to zero. Does that make any sense?
p of flip a coin is 0.5, X=112, mean(u)=n*p=0.5*200. You
can calculate SD using sigmaˆ2=np(1-p) and z=(x-u)/sd,
and use Standard Normal Distribution Table.
What Data to Create Analytics Of?

27. Content-Related Network
Content-related network included
fewer learners but with higher
degree and edge weights
Non-content interactions had
shorter threads with less repeat
participants, simpler topics and
fewer social presence cues
Wise, A. F., & Cui, Y. (2018). Learning communities in the crowd: Characteristics of content related
interactions and social relationships in MOOC discussion forums. Computers & Education, 122, 221-242.
Non-Content Network
What Data to Create Analytics Of?
MOOC Statistics Discussion

28. Content-Related Network
Content-related network included
fewer learners but with higher
degree and edge weights
Non-content interactions had
shorter threads with less repeat
participants, simpler topics and
fewer social presence cues
Only content interactions were
predictive of final grades
Wise, A. F., & Cui, Y. (2018). Learning communities in the crowd: Characteristics of content related
interactions and social relationships in MOOC discussion forums. Computers & Education, 122, 221-242.
Non-Content Network
What Data to Create Analytics Of?
MOOC Statistics Discussion

29. Constructs to Clicks
Connect constructs to clicks to create cogent analytics and develop metrics
to refine and expand collaborative constructs
Not just about how calculation done but what data is included / excluded
Consider individual- & group-level constructs + relationships between them
Integrate analytical and interpretive methods to connect high-level
abstractions with detailed process accounts
Paulus T. M. & Wise, A. F. (2019). Researching learning, insight, and transformation in online talk.
New York, NY: Routledge.

30. 2.
Making analytics
actionable (really)
Collaborative Learning Analytics

31. Analytics for…
1. What? The relative balance of
technology and human agency
2. Who? Support for activity at different
levels (group, individual, collective)
3. When and How? Iterations of refining
collaborative learning efforts

32. Adaptive Team Systems
Algorithmically Initiated Changes?
“Intelligent technologies….assess the current state
of the interaction to provide a tailored pedagogical
intervention” (to group configurations, interactions or
understanding) Soller, 2015
“The computer environment should not be providing the
knowledge and intelligence to guide learning, it should
be providing the facilitating structure and tools that
enable students to make maximum use of their own
intelligence and knowledge” Scardamalia et al., 1989

33. Adaptive Team Systems
Algorithmically Initiated Changes!
Rummel, N., Walker, E., & Aleven, V. (2016). Different futures of adaptive collaborative
learning support. International Journal of Artificial Intelligence in Education, 26(2), 784-795.
We are not pre-destined to a “dystopian” future in which
artificial intelligence based support for collaboration is
reactive, rigid, and robs learners (and teachers) of agency
Instead, we need a vision for a more “utopian” future in which
adaptive support is provided in a responsive, nuanced and
flexible way to customize, adapt or fade scripts over time
Are these temporary scaffolds or performance support?

34. Adaptive Team Systems
Algorithmically Initiated Changes…
Wise, A. F. Vytasek, J. M., Hausknecht, S. N. & Zhao, Y. (2016). Developing learning
analytics design knowledge in the “middle space”: The student tuning model and align design
framework for learning analytics use. Online Learning, 20(2), 1-28.
Need to imagine what productive collaboration between
people and adaptive systems (agents or not) looks like
Knowing when to disagree with analytics (and being
empowered to do so) is both an important competence
to build, and a more effective pedagogic strategy than
attempting to develop analytics that are “perfect”
Kitto, Shum & Gibson, 2018

35. Adaptable Team Systems
User Initiated Changes
Building on existing traditions of group awareness tools
New generation of collaboration dashboards
Who are we expecting to interpret and act on this
information, when and how?

36. Liu, A. L., & Nesbit, J. C. (2020). Dashboards for Computer-Supported Collaborative Learning.
In Machine Learning Paradigms: Advances in Learning Analytics (pp. 157-182). Springer.
Adaptable Team Systems
Extracted Analytics

37. Marbouti, F. & Wise, A. F. (2016) Starburst: A new graphical interface to support productive engagement with
others’ posts in online discussions. Educational Technology Research & Development, 64(1), 87-113.
Zhang, J., Tao, D., Chen, M. H., Sun, Y., Judson, D., & Naqvi, S. (2018). Co-organizing the collective
journey of inquiry with Idea Thread Mapper. Journal of the Learning Sciences, 27(3), 390-430.
Adaptable Team Systems
Embedded Analytics

38. Individual Learners
Small Groups
The Collective
Adaptable Team Systems
Multiple Levels of Support & Action

39. Wise, A. F. Vytasek, J. M., Hausknecht, S. N. & Zhao, Y. (2016). Developing learning
analytics design knowledge in the “middle space”: The student tuning model and align design
framework for learning analytics use. Online Learning, 20(2), 1-28.
Relative to
Self
Relative to
Others
Absolute
Levels
With
Self
With
Peers
With
Instructors
Adaptable Team Systems
Intentional Iterative Refinement

40. Productive Process
Indicators
Purpose of
Team Activity
Learning Analytic
Metrics
Articulating one’s
ideas, being exposed
to the ideas of
others, negotiating
differences in
perspective
Attending deeply
to a spectrum of
others’ ideas, and
contributing
comments that
are responsive
and rationaled,
Percent of posts
read introduced
as a metric that
has clear
meaning in the
context of the
activity
Adaptable Team Systems
Intentional Iterative Refinement

41. Individuals
Wise, A. F., Zhao, Y. & Hausknecht, S. N. (2014). Learning analytics for online discussions:
Embedded and extracted approaches. Journal of Learning Analytics, 1(2), 48-71.

42. Small Groups
van Leeuwen, A., Rummel, N., Holstein, K., McLaren, B. M., Aleven, V., Molenaar, I., ... & Segal, A. (2018).
Orchestration tools for teachers in the context of individual and collaborative learning: what information do
teachers need and what do they do with it?. Proceedings of ICLS 2018.

43. Collective
Always Same
People
Always Different
Actual Class
Avg Degree = 3
Modularity = .81
Avg Degree = 10
Modularity = .14
Avg Degree = 9
Modularity = .27

44. Actionable Analytics
Design analytic systems that support rather than supplant learner agency
Consider targets and action at individual, group and collective levels
Choose adaptive or adaptable systems and embedded or extracted
solutions to meet specific learning needs
Plan for (and document) a process of iterative improvement
Q: Does responsive feedback relax or fortify predetermination?

45. Towards Collaborative
Learning Analytics
Opportunities, Challenges and
Tensions at the Intersection of
CSCL and LA
Alyssa Friend Wise
Director, NYU Learning Analytics Research Network
Associate Professor of Learning Sciences & Educational Technology
New York University