Citation analysis of learning analytics field from conference proceedings and Journal special issues

1. S. Dawson
D. Gasevic
G. Siemens
S. Joksimovic
Current State and Future Trends:
A citation network analysis of the Learning
Analytics Field

2. Goal
• Citation analysis and structural mapping to
gain insight into the influence and impact
within LA
– a snapshot of LA through analysis of articles and
citations (LAK conferences and special issues)

3. Context
• Although much potential and excitement:
– to date LA has served to identify a condition, but
has not advanced to deal with the learning
challenges in a more nuanced and integrated
manner

4. Aim
• Identify emergence of influential trends and
hierarchies in the field
• Commencement point (Leah):
– a foundation for future work
– identify promising areas of research
– Identify under represented disciplines
– Improve integration across disciplines and theory
and practice

5. Context
• Learning analytics:
– has emerged as a field (maturation)
– multi-disciplinary
– often mis-represented and poorly understood
• (Academic analytics; business intelligence; assessment
analytics; social analytics; web analytics; educational
data mining)

6. Approach
• Bibliometrics measure the impact/influence of
an author or article using various citation
analyses
• Garfield 1955 (Impact)
Ding, Y. (2011). Scientific collaboration and endorsement: Network analysis of
coauthorship and citation networks, Journal of Informetrics, 5(1), 187-203

7. Data
• LAK11, 12, 13
• Special issues – ETS, JALN, ABS
• Data analysis
– Citation counts
– Author/citation network analysis
– Contribution type
– Research methods
– Author disciplinary background

8. Approach
• Citation and author networks:
– Identify the prominent research
– Identify linkages between disciplines and authors
– Identify diversity of research genres

9. Citation analysis
• The use of citations long been used to
measure impact
– Core output of research is publications
– As research grows, output (papers) further build
on other associated works (citations)
– “quality” as quantity of citations
• Identify areas of prominent research activity

10. Citation analysis
• Criticisms
– Cronyism
– Self citations
– “Rich get richer”

11. By the way some great refs
Gašević, D., Zouaq, A., & Janzen, R. (2013). “Choose Your
Classmates, Your GPA Is at Stake!” The Association of Cross-
Class Social Ties and Academic Performance. American
Behavioral Scientist 57 (10), 1460-1479
Siemens, G. (2013). Learning Analytics The Emergence of a
Discipline. American Behavioral Scientist 57 (10), 1380-1400
Dawson, S., Tan, J., & McWilliam, E. (2011). Measuring creative
potential: Using social network analysis to monitor a learners'
creative capacity. Australasian Journal of Educational
Technology 27 (6), 924-942

12. Citation analysis

13. Citation analysis
• Ok Some faults – broadly accepted that
quality is linked to the number of citations

14. Citation analysis
• Number of citations (Highly cited articles)
– Predominately conceptual and opinion papers
(e.g. Educause)
– Methods (Wasserman and Faust)
– Few empirical studies
• Citation and author networks:
– Illustrate linkages and disciplines

15. Example author networks
Paper 1: Liz, Julie, Sara
Sara
JulieLiz

16. Example author networks
Paper 1: Liz, Julie, Sara
Sara
JulieLiz
2 Degrees

17. Example citation network
Paper 1: Liz, Julie, Sara
Cites: Liam & Luka
Sara
Julie
Liz
Liam
Luka

18. Citation network
LAK conference
Journals

19. Citation networks
• Citation network moderate level of clustering
– Consistent across LAK proceedings
– Few strong connections?
– Degrees low – indication of diverse and
inconsistent literature sources
– Degrees (increasing) from LAK11 to 13

20. Author networks (LAK)

21. Author networks (Journal)

22. Author networks
• Author networks – small cliques with few
highly connected nodes
• For an interdisciplinary field still largely
disciplinary clustered

23. Paper classification
• Schema from Info Systems (6 categories)
1. Evaluation research
– (e.g. case study empirical)
2. Validation research
– (e.g. testing theory/ method/ solution empirical)
Glass, R.L., et.al, 2002. Research in software engineering: an analysis of the
literature. Information and Software Technology 44, 8, 491-506

24. Paper classification
3. Solution proposal
(solution/ technique to address an issue)
4. Conceptual proposal
(e.g. frameworks)
5. Opinion
(well its my opinion/argument)
6. Experience
(Let me tell you a story)
7. Panel/workshop

25. Author Background
Extensive search – if home dept not listed in
author details

26. Paper classification (LAK)
• Dominated by evaluations of research
(journals)
• Proposal of solution dominates conference

27. Paper classification (Journal)
• Dominated by evaluations of research
(journals)
• Proposal of solution dominates conference

28. Paper classification
• Dominated by computer science (LAK)
• Greater number of education researchers in
journals
– Reflection of special issues
– Reflection of priority sites for publications
• Largely conceptual and opinion publications

29. Methods classification
• Qualitative
• Quantitative
• Mixed methods
• Other

30. Methods classification (LAK)
• Other dominates – reflects high number of
opinion and conceptual papers

31. Methods classification (Journals)

32. Methods classification
• Other dominates – reflects high number of
opinion, experience and conceptual papers

33. Conclusions
• The field is in its infancy
– Citations still predominately opinion and
definitional pieces
– Clustering and degrees
– Few number of empirical studies cited but this is
growing
• Mature fields greater examples of validation
research and importantly critiques of studies

34. Conclusions
• Computer scientists dominate LAK
proceedings
– Need to look at how other voices are heard
• Education research dominates Journals
– Reflection of broader priorities?

35. Conclusions
• Early work need to extend
• Structural mapping and citation analyses more
common and more sophisticated.
• Raise awareness
– Inform practice
– Build connections
– Foster further empirical work

36. Conclusions
• Understanding our field we can better
advance our field.
• Question: To what extent can we use these
analyses to architect the development of the
field?

37. Questions
• Next steps:
–Broader scope (extend network
analyses)
–Keyword clustering
–Citation location
–Incorporate multiple citations/ paper

38. Questions
• To what extent can we use these
analyses to architect the development
of the field?
• shane.dawson@unisa.edu.au
• dgasevic@acm.org
• gsiemens@gmail.com
• sreckojoksimovic@gmail.com