Workshop on Educational and Social Sciences, 19-21 Jul, at the National University of Singapore. Tutorial.

1. LearnSphere and DataShop
Enabling Data Fusion across
Diverse Data Sources
John Stamper
Assistant Professor
Human-Computer Interaction Institute
Carnegie Mellon University

2. LearnSphere
“a community software infrastructure around
the analysis of educational data that supports
sharing and collaboration across the wide
variety of educational data”
2

3. 3

4. 4
http://learnsphere.org

5. Data Silos => Data Integration
Many paradigms of data-
driven education
research differ in
● data types
● time scale
● research goals
Disciplinary silos are
fostered by differences
Data infrastructure for
analytics across these
Ultimate goal: Produce
discoveries not
possible within current
silos
5

6. Distributed deployment and
storage
Central LearnSphere portal
Multiple “myLearnSphere”
installations
● Individual installations
curate their own data or
replicate to central
repository
● Outside researchers can
identify existing datasets
through metadata
provided by local
versions
6

7. Learning Analytics Workflow Authoring
Environment (Tigris)
7

8. • Central Repository
– Secure place to store & access research data
– Supports various kinds of research
• Primary analysis of study data
• Exploratory analysis of course data
• Secondary analysis of any data set
• Analysis & Reporting Tools
– Focus on student-tutor interaction data
– Data Export
• Tab delimited tables you can open with your favorite spreadsheet
program or statistical package
• Web services for direct access
DataShop
8

9. Repository
• Allows for full data management
• Controlled access for collaboration
• File attachments
• Paper attachments
• Great for secondary analyses
How much data does DataShop have?

10. How big is DataShop?
10
Domain Files Papers Datasets Student Actions Students Student Hours
Language 64 14 123 13,369,151 15,125 58,533
Math 283 83 417 140,002,452 170,403 317,008
Science 113 18 251 29,787,644 64,569 90,547
Other info 85 22 187 24,746,456 51,765 100,617
Unspecified 120 0 458 40,923,767 53,713 107,393
Total 665 137 1,436 248,829,470 355,575 674,100
As of July 2017

11. What kinds of data?
• By domain based on studies from the Learn Labs
• Data from intelligent tutors
• Data from online instruction
• Data from games and simulations
The data is fine grained at a transaction level!

12. Web Application

13. • Explore data through the DataShop tools
• Where is DataShop?
– http://pslcdatashop.org
– Linked from DataShop homepage and learnlab.org
• http://pslcdatashop.web.cmu.edu/about/
• http://learnlab.org/technologies/datashop/index.php
• http://learnsphere.org
Getting to DataShop
13

14. Creating an account
• On DataShop's home page, click
“Login". Complete the form to create your
DataShop account using InCommon or Google
ID.
• If you’re a student/staff/faculty of a university, login with
InCommon to create your account, or use the Google login.
14

15. Getting access to datasets
• By default, you will have access to the public
datasets.
• For access to other datasets, you can request
access from dataset
15

16. DataShop Terminology
• Problem: a task for a student to perform that
typically involves multiple steps
• Step: an observable part of the solution to a
problem
• Transaction: an interaction between the
student and the tutoring system.

17. DataShop Terminology
• Observation: a group of transactions for a particular
student working on a particular step.
• Attempt: transaction; an attempt toward a step
• Opportunity: a chance for a student to demonstrate
whether he or she has learned a given knowledge
component. An opportunity exists each time a step is
present with the associated knowledge component.

18. DataShop Terminology
• KC: Knowledge component
– also known as a skill/concept/fact
– a piece of information that can be used to
accomplish tasks
– tagged at the step level
• KC Model:
– also known as a cognitive model or skill model
– a mapping between correct steps and knowledge
components

19. Example
19

20. Base1 6
Base2
Base3
ExpandedPower1 100,000,000
ExpandedPower2
ExpandedPower3
Exponent1 8
Exponent2
Exponent3
GeneralHelpGoal
Node
Multiplier1 6
Multiplier2
Multiplier3

21. Transactions Student-Steps
Enter 8 in Multiplier1 Multiplier1
Ask for hint on next step
ExpandedPower1
Ask for hint
Enter 10,000 in ExpandedPower1
Enter 100,000 in ExpandedPower1
Enter 8 in Base1
Multiplier ExpandedPower Base Exponent
Multiplier1
Multiplier2
Multiplier3
ExpandedPower1
ExpandedPower2
ExpandedPower3
Base1
Base2
Base3
Exponent1
Exponent2
Exponent3
Enter 6 in Exponent1
Enter 5 in Exponent1
Base1
Exponent1
8 100,00010,000
8
65

22. Transactions Student-Steps
Multiplier1 UpdateTextField 8 Multiplier1 Multiplier 1
HintButton ButtonPressed HintRequest
ExpandedPower1 Exp.Power 1
HintButton ButtonPressed HintRequest
ExpandedPower1 UpdateTextField 10,000
ExpandedPower1 UpdateTextField 100,000
Base1 UpdateTextField 8
Multiplier ExpandedPower Base Exponent
Multiplier1
Multiplier2
Multiplier3
ExpandedPower1
ExpandedPower2
ExpandedPower3
Base1
Base2
Base3
Exponent1
Exponent2
Exponent3
Exponent 1 UpdateTextField 6
Exponent1 UpdateTextField 5
Base1 Base 1
Exponent1 Exponent 1
8 100,00010,000
8
65
KC OpportunitySelection Action Input Step

23. Transactions Student-Steps
Multiplier2 UpdateTextField 8 S1 Multiplier1 Multiplier 1
S1 ExpandedPower1 Exp.Power 1ExpandedPower2 UpdateTextField 100,000
ExpandedPower2 UpdateTextField 1,000,000
Base2 UpdateTextField 8
Multiplier ExpandedPower Base Exponent
Multiplier1
Multiplier2
Multiplier3
ExpandedPower1
ExpandedPower2
ExpandedPower3
Base1
Base2
Base3
Exponent1
Exponent2
Exponent3
Exponent 2 UpdateTextField 6
S1 Base1 Base 1
S1 Exponent1 Exponent 1
8 1,000,000100,000
8
6
Selection Action Input Student Step
S1 Multiplier2 Multiplier 2
S1 ExpandedPower2 Exp.Power 2
S1 Base2 Base 2
S1 Exponent2 Exponent 2

24. Learning Curves
24
Visualizes changes in
student performance over
time
Time is represented on the x-
axis as ‘opportunity’, or the # of
times a student (or students)
had an opportunity to
demonstrate a KC
Hover the y-axis to change the
type of Learning Curve.
Types include:
• Error Rate
• Assistance Score
• Number of Incorrects
• Number of Hints
• Step Duration
• Correct Step Duration
• Error Step Duration

25. Learning Curves: Drill Down
25
Click on a data point to
view point information
Click on the number link to
view details of a particular drill
down information.
Details include:
• Name
• Value
• Number of Observations
Four types of information
for a data point:
• KCs
• Problems
• Steps
• Students

26. Learning Curve: Latency Curves
26
For latency curves, a
standard deviation
cutoff of 2.5 is applied
by default.
The number of included
and dropped
observations due to the
cutoff is shown in the
observation table.
Step Duration = the total length of time spent
on a step. It is calculated by adding all of the
durations for transactions that were attributed
to a given step.
Error Step Duration = step duration when first
attempt is an error
Correct Step Duration = step duration when the
first attempt is correct

27. Dataset Info: KC Models
27
Handy information displayed for
each KC Model:
• Name
• # of KCs in the model
• Created By
• Mapping Type
• AIC & BIC Values
Toolbox allows you
to export one or more
KC models, work with
them, then reimport
into the
Dataset.
DataShop generates two
KC models for free:
• Single-KC
• Unique-step
These provide upper and lower
bounds for AIC/BIC.
Click to view
the list of KCs
for this model.

28. Dataset Info: Export a KC Model
28
Export multiple models at once.
Select the models you wish
to export and click the
“Export” button.
Model information as well as
other useful information is
provided in a tab-delimited
Text file.
Selecting the “export”
option next to a KC Model
will auto-select the model
for you in the export
toolbox.

29. Dataset Info: Import a KC Model
29
When you are ready to import,
upload your file to DataShop for
verification.
Once verification is successful,
click the “Import” button.
Your new or updated model will
be available shortly (depending
on the size of the dataset).

30. A KC model produces a
learning curve
Without decomposition, using
just a single “Geometry” skill,
Is this the correct or “best”
cognitive model?
no smooth learning curve.
a smooth learning curve.
But with decomposition,
12 skills for area,
(Rise in error rate because
poorer students get
assigned more problems)

31. LearnSphere Workflow: Integrating data silos
• Share methods
– Curation: Variable extraction, standardizing formats
– Analysis: Statistics, machine learning
– Visualization & reporting
• Share data
• Multi-language integration
– Components written in any language: R, Python, Java,
C++, Matlab, etc.
• Non-programmers can recombine components to execute
different analyses
• Easily replicate others’ analysis workflows on new datasets

34. http://pslcdatashop.org
Questions?
john@stamper.org
datashop-help@lists.andrew.cmu.edu
34