Presented at the Third Conference on Learning Analytics and Knowledge (LAK 2013).

1. Third Conference on Learning Analytics and Knowledge
(LAK 2013)
April 8-12, 2013
Leuven, Belgium
From Micro to Macro -
Analyzing Activity in the ROLE Sandbox
Dominik Renzel, Ralf Klamma
RWTH Aachen University
Advanced Community Information Systems (ACIS)
Aachen, Germany
{renzel,klamma}@dbis.rwth-aachen.de
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
1 These presentation slides by Dominik Renzel are licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

2. Advanced Community
Information Systems (ACIS)
Responsive
Web Engineering Community
Web Analytics
Open
Visualization
Community
and
Information
Simulation
Systems
Community Community
Support Analytics
Lehrstuhl Informatik 5
Requirements
(Information Systems)
Prof. Dr. M. Jarke
2
Engineering

3. Motivation
 Increasing adoption of distributed learning services on the Web
 Large-scale Web-based learning platforms (MOOC, PLE, hybrids)
 Monitoring: unprecedented insights into learner behaviour
– Proprietary techniques & data models for specific scenarios (bias/limitation)
– New “standards“ for learner behaviour analysis on high semantic level
 But: Learner behaviour evolving & inherently hard to model!
 Our proposal: step back to existing Web logs
 Standardized by-product of regular Web service operation
 World-wide adoption
 Lowest possible degree of bias
 Generic low-level semantics easily liftable to higher-level semantics
 Analysis on multiple ecosystem levels from “micro to macro“
Lehrstuhl Informatik 5
(Information Systems)  Highest level of data interoperability (cross-service analysis)
Prof. Dr. M. Jarke
3

4. Focus & Level of Analysis –
Bronfenbrenner‘s EST applied to LA
5 nested & interrelated ecosystems
around student:
 Microsystem
(direct relation to peer/group)
 Mesosystem
(entirety of microsystems + relations)
 Exosystem
(networks influencing individual)
 Macrosystem
(entirety of societal relations incl. norms, rules, etc.)
 Chronosystem
(temporal dimension of development)
 Existing work: often focus on specific system or subsets only…
 Any comprehensive LA framework…
 should allow analysis & focus on all levels
 may not neglect micro data
Avoid “washing out“ anomalies!
Lehrstuhl Informatik 5  must apply aggregation carefully
(Information Systems)
Prof. Dr. M. Jarke  should work with historical & real-time data Support targeted & timely intervention!
4

5. Data Interoperability
 Proprietary data “standards“ for monitoring learner activity in LA
– High-level semantics for convenient analysis from the beginning
 danger of losing anomalies, especially interesting in LA!
– Long standardization process with no guarantee for adoption
 risks: repeated costly changes to LA installations necessary!
– Use of competing/incompatible data formats for different LA frameworks
 cross-service analysis complicated/unfeasible!
 Why not build on existing Web standards?
– Web logs (+ page tagging) provably powerful tools in Web Analytics
Lehrstuhl Informatik 5
(Information Systems)
– Profit from genericity & widespread use also in LA
Prof. Dr. M. Jarke
5 – The Web works incredibly well with these standards! LA might, too…

6. Opacity & Uncertainty
 LA inherently suffers from uncertainty regarding user identity
 Web Analytics techniques can improve, but never solve the problem!
 Low-level Web log analysis
 Not perfect, but always available
 Allows sufficiently precise identification of structural/behavioral patterns
Lehrstuhl Informatik 5
(Information Systems)  Allows inferences about opaque parts
Prof. Dr. M. Jarke
6

7. Real-time Web Log Processing Pipeline
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
7

8. Example: Adding widget to
Widget-based PLE (ROLE Sandbox)
Example Web log entry:
<IP> <DATE> POST /space/<space-id>:;tool=<widget-url>
Lehrstuhl Informatik 5
Geo Location: Widget Metadata:
(Information Systems)
Prof. Dr. M. Jarke Longitude/Latitude,City,Region, Country Domains, Activities
8

9. Analysis Techniques &
Benefits for Learning Stakeholders
 Analysis techniques possible with simple Web log data
– Which learners/learner communities interacted how with which
tools, resources in which context where and when?
– Single operations  micro-level analysis
– Appropriate aggregation  up to macro-level analysis
– Relations in log data  actor networks  SNA
– Temporal information  time series analysis
– Semantic enrichment from external sources  a lot more!!!
 Benefits for learning stakeholders
– Learning community awareness (e.g. by visualization)
Lehrstuhl Informatik 5
– Recommendations (e.g. tools, persons, resources)
(Information Systems)
Prof. Dr. M. Jarke
9

10. Conclusions & Future Work
LA frameworks for analyzing Web-based learning services to…
– Prevent bias/limitation by relying on generic Web standards
– Focus & level of analysis to cover micro to macro + chrono
– Enable large-scale cross-service analysis by data interoperability
– Combine historical & real-time information for intervention planning
– Fight uncertainty, esp. regarding learner identity
 Argument: Web logs stay superior to “new standards“
 Not without limitations, but always available without further instrumentation
 Web log processing pipeline incl. data enrichment for LA simple to setup
 Multiple analysis techniques applicable and rich information derivable
Next steps:
 Carry out analysis on ROLE Sandbox dataset recorded from Mar 2012 - now
 >8 Mio API requests from > 3000 IPs, >500 Widgets, >1300 PLE Spaces,
>3700 shared resources
Lehrstuhl Informatik 5
 Apply approach in other scenarios & projects
(Information Systems)
Prof. Dr. M. Jarke
10

11. Questions?
Contact: renzel@dbis.rwth-aachen.de
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
11