The document outlines a data-driven learning strategy emphasizing the importance of big data in enhancing educational practices and personalized learning experiences. It highlights the need for improved evidence frameworks to analyze diverse and granular educational data from digital learning systems, alongside collaborative efforts between researchers and educational institutions. Additionally, it discusses the role of technologies like xAPI in tracking learning activities and the importance of community-driven standards for effective data interoperability in educational settings.

1. Data-Driven Learning Strategy
Jessie Chuang
Classroom Aid Inc.

2. Data is
the only hard evidence of what works in
education,
and
the compass for personalized learning journey.

3. Data, Data, Data
Amazon: data driven engine accounts for 1/3 of
all sales
Netflix: recommendation engine accounts for 3/4
of all new orders
How about learning and training segment?

4. Evidence Framework
A report by U.S. Dep. of Ed.
- discusses the promise of sophisticated digital learning systems for collecting and
analyzing very large amounts of fine-grained data (“big data”) as users interact
with the systems.
- describes an iterative R&D process, with rapid design cycles and built-in
feedback loops—one familiar in industry but less so in education (however, the
report provides numerous examples of applications in education).

5. Two critical steps - 1
● Education must capitalize on the trend within
technology toward big data.
● New types of data are becoming available.
● broad, static categories(demographic, end-of-year grades and test
scores) => more dynamic, fine-grained data in the process of learning
● data in a single context => linked across different parts of learning
● collected through different methodologies and reported in isolation =>
aggregating evidences from different data sources

6. Two critical steps - 2
● A revitalized framework for analyzing and using evidence
○ Gold standard evidence - slow and expensive, poor matches to the
rapid pace of digital development practices
○ An evidence framework should help educational stakeholders align their
methods of obtaining evidence with their goals, the risks involved, the
level of confidence needed, and the resources available.
Confidence
Risk
The need of evidence
before implementation
collection of data should be
an ongoing task

7. Challenges addressed by evidence approaches
● Making sure learning resources promote deeper learning
● Building adaptive learning systems that supports
personalized learning
● Combining data to create support systems more
responsive to student needs
● Improving the content and process of assessment with
technology
● Finding appropriate learning resources and making
informed choices

8. Evidence Approaches w/i Great Potential
1. Educational data mining and learning analytics applied to data gathered from digital
learning systems implemented at scale
2. Rapid A/B testing conducted with large numbers of users within digital learning systems
(example: Khan Academy)
3. Design-based implementation research (DBIR) supported by data gathered from digital
learning systems (collaborative design from developers, researcher, teachers)
4. Large datasets of different types from multiple sources, combined and shared across
projects and organizations
5. Technology-supported evidence-centered design (ECD) of measures of student learning
6. Data gathered from users about a learning resource, how users have used it and their
experiences using it. => Example: enable “Rapid Prototyping” by making early version of a
product freely available online, collect data and mine data for insights; ratings and reviews
from users …...

9. What Works Clearinghouse
(by U.S. Department of Education)
- “The high volume of research on different programs, products, practices, and
policies in education can make it difficult to interpret and apply the results. We
review the research. Then, by focusing on the results from high-quality research,
we try to answer the question “What works in education?”
- The goal is to provide educators with the information they need to make
evidence-based decisions.

10. iZone - Researchers and Schools collaborate on
an Education Test Bed
● Partnership from 250 schools, Research Alliance (based at NYU), EdSurge,
ChallengePost, and IDEO
● Supporting from U.S. Dep. of Education, Bill&Melinda Gates Foundation, New
York City Dep. of Education
● To support developers in rapidly developing and testing selected technology
based instructional supports and featuring test results on EdSurge
● To address unmet needs identified by a diverse group of school stakeholders
● Launch prize competitions for developers, winners will be invited to beta-test
products in iZone classrooms
● Researcher have access to resources necessary to a given study; test bed data
are available for secondary analysis to continue relevant study (also to
develop analytic capability and let good algorithms bubble up)
● The collaboration will produce Consumer Reports-style guide for learning tech.
● Silicon Valley also initiated an iZone to create a similar ecosystem

12. collegestats.org

13. Learning Analytics Applications
● Learner modeling: model a learner’s knowledge, behaviors, motivation,
experience…
● Learner profiling: cluster users into similar groups
● Domain modeling: decompose content into components and sequences
● Effectiveness: test learning principles, pedagogies etc.
● Trend analysis: understand changes over time
● Recommendation and adaptation: suggest resources and actions to learners
or instructors, adapt systems to learners...

14. Research examples
Combine observation data with click streams to detect
boredom, frustration, gaming the system…
● Off-task behaviors patterns can be identified like
completing module too fast or too slow …
● Evidences of gaming the systems like systematic and rapid
incorrect answers, systematic guessing …
Experience API, xAPI, can give us more because …

15. Learner
*Motivator
*Navigator (content brokering)
*e-Portfolio(or digital badge backpack)
inform iterate
co-design
Data Analyst Learning Designer
Standards
Interoperability
Analysis (data mining)
Learning theories
Psychology
Design (experience design, UI)
Subject Matter Expertise
integrate
(developers are constructors)
*Connecting Learner Community
*Learner Attribute (style)
*Narrator (learner created
context, reflection)
Gamified layer
on data from
across platforms
Big Data
[Data-Driven Learning Strategy]

16. Intelligent learning environment is to match a
learner with the right content at the right time.

17. About Learner Profiling
Image credit: DARPA

18. Recording Learning Events
Learning happens in interactions:
openclipart.org
Contents: Courses, Books,
Web pages, Games, AR ….
Instructors, Peers, Experts….
Activities(making, exercises,
researching, online, offline ….)
Learner

19. Recording Learning Events
Social Learning
Group Learning
image credit: Search Engine People Blog

20. Learning
is
different

22. image credit: “A New Architecture for Learning”, published by Educause

23. xAPI tracking all kinds of learning experiences

24. The xAPI specification has two primary parts
1. defines the syntax of the xAPI data format
a. the vocabularies should be community-driven
b. all activities and context can be tracked
c. any enabled application/device can send statements
2. defines the characteristics of “learning record stores” (LRS)
- a crucial component of xAPI
a. data can be exchanged between LRSs (set free from LMS)
b. learner can have life-long “personal learning locker”
c. LRSs need to validate xAPI statements

25. Social Coaching
Project Learning Mobile
Apps
LRS Learning data
is sent to LRS
Other
activities
Course Webpage Game Simulator

26. LMS LRS
LRS
Reporting
Tool
Learning records
can be delivered
to LMSs，LRSs
or Reporting
Tools.

27. Recording Learning Events
ActivityStreams: stream of activity data statements,
borrowed from social analytics
This is only the basic idea. Crafting the statements with more
context and related information is necessary to support
analytics and reporting.

28. 11 Attributes in xAPI Data Format
★ Unique Identifier
★ Actor (required)
★ Verb (required)
★ Object (required)
★ Result
★ Context
★ Timestamp
★ Stored (internal recording
timestamp)
★ Authority
★ (Protocol) Version
★ Attachments
All information in XAPI statements can
be separated into :
● meta-data,
● descriptive information, and
● complementary data.

29. Syntax-1
Actor:
Agent (= persona) or group (multiple IDs allowed)
Verb:
ID = an IRI(URL) = a specific semantic meaning +
human readable display
Object:
an agent, a group, a statement or an activity(most common)

30. Syntax-2
Activity:
ID = an IRI(URL) = with a specific boundary (granularity)
Definition:
Name
Description
objectType
Extentions (useful to customize reporting)

32. Syntax-3
Result:
Score
Success
Duration
Completion
Response (learner’s response to the experience)
Extentions

33. Syntax-4
Context:
Registration (differentiate multiple attempts)
Instructor
Team
ContextActivities (parent, grouping, category, other - like related lesson)
Revision
Platform
Language
Statement (refer to one other statement for a whole experience)
Extentions

34. Context, Context, Context
The XAPI differentiates between the core context and the wider context.
The core context includes the instructor(s), the direct peers involved in an activity
(team), the learning environment (platform), the language that was used in the
performance, and a framing statement for an activity (e.g., the course that relates
to the activity).
The extended context includes a set of data-records about the wider context of a
learning activity. This wider context is not explicitly specified and can include the
location of the learner, the wider (social) relations, the duration of an activity,
environmental factors (e.g., temperature or noise level) etc. The format and the
content of the wider context is specific to the AP and is not subject to the
interoperability of the data format.

35. Ask the right questions!

36. LRS
An LRS is defined by two interfaces:
● Statement interface (statement API)
● Document interface - this interface handles three types of documents
● State interface (state API)
● Activity profile interface (activity API)
● Agent profile interface (agent API)
The LRS is responsible for
1. validating that the system sending data is authorized,
2. checking that the data being sent is xAPI-compliant,
3. storing the data properly,
4. making that data available to any other authorized system or activity provider
when asked.

37. Advanced Applications of xAPI
- Data Transfer based on RESTful HTTP w/i LRS
● Agent Profile API
○ personal info., learner profile and modeling, user settings, learning
journal, career plan & goal
○ an integrated picture of a learner activities across systems and devices
with multiple identities
● Activity Profile API (for activity provider)
○ interactions between learners (collaboration, social or competition)
○ learning planning tool (access to or update the LRS internal definition of
a given activity id, even before the activity sends any statement)
● State API
○ persist state across devices
● Authentication services, querying services, visualization services, and
personal data services are some examples.

38. xAPI tracking is...
Semantic
Contextualized
From any device and sensor
LRS frees the learning data so they can be put together, analyzed, modeled,
reused, carried with learners and accumulated life-long

39. xAPI + Open Badges => Learner ePortfolio
Both are representing learner data by exploiting HTTP, JSON,
and REST - simple, lightweight method that lowers entry
barrier for developers.
Together, they offer a new way to think about constructing
interoperable learner model data!

40. JSON vs XML
● JSON is bandwidth-non-intensive
● JSON is better adapted (than XML) to devices with limited capabilities such as
smart things
● JSON possesses a very limited set of data types. Restricting itself to primitive
data types makes it deeply and immediately interoperable with pretty much
any programming language that exists out there.
● JSON is a better data exchange format, XML is a better document exchange
format
● JSON is a preferred format in NoSQL database - data not seamlessly conform
to a columnar/relational model. Since JSON objects may be heterogeneous in
terms of number and types of fields, this allows for tremendous flexibility in
storing and retrieving objects as compared to relational databases.
(Should you like to know more about JSON, here is a very simple interactive tutorial. )

41. Bootstrapping Learner Model Data
● Making data contribution as easy as possible
● Bootstrapping heterogeneous learner model
data
○ the raw experience data stored in a LRS referenced by an Open Badge,
could be analyzed to perform an application’s own interpretations of that
evidence, as suggested by Carmagnola, Cena, and Gena (2011)
○ Guo & Greer, 2007;
○ Tiroshi, Kuflik, Kay & Kummerfeld, 2011 ) summarized various
boostrapping methods
○ ADL developed an open source project/resource called “lr-data”
(Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)

42. Sharing and Analysis of Data
In the case of sharing and making sense of learner data, a similar model can be
followed where any contributing application (e.g., an ITS or game) could easily
publish activity data to a LRS, Open Badges to a learner’s badge backpack, or
other learner model data to their shared learner profile as JSON over a RESTful
HTTP connection.
The LRS, badge backpack, or learner profile can exist anywhere (e.g., an employer
organization, a commercial provider, or even self-hosted by a learner) – all the
contributing applications need to know are the URLs and the learners credentials.
If a learning application wants to make sense of all this data, it too just needs
the URLs and the learner’s credentials to get started. The learner could set
permissions concerning what applications can access their data.
(Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)

43. TLA Services
TLA will also include services for managing learner profiles. Open Badges can be
referenced by learner profiles, which will likely contain other learner data such as
goals, reflection, etc.
The TLA will also include services for creating and accessing competency
definitions to serve as a common way to reference educational standards, learning
objectives, and competency definitions through web APIs
(Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)

44. The whole picture =
Training and Learning Architecture(TLA)
● ePortfolio
● Learner modeling
● Machine readable
● Competency
standards
● Knowledge map
● Standard
alignment
● xAPI COP
● Common
vocabulary
● Learning
Design
● Sharing of
metadata &
paradata (LR)
● Re-usability
● Semantic analysis

46. A taxonomy of education standards
Redd, Brandt

47. Education Standards for Interoperability
❏ Academic standards
❏ Data Standards (consistent definition)
❏ Student data
❏ Data Dictionary (data element definition)
❏ Logical Data Model (entity definition)
❏ Educator data
❏ Content data (metadata, paradata)
❏ xAPI vocabulary/activity profile
CEDS

48. Major Learner Data Categories
● Educational records
● Competencies(skills, knowledge, abilities,
outcomes…) and domain learning objectives
● Data in affective, motivational(disposition) and
social dimensions
● Data for learning style modeling
● Data for pedagogical purposes

50. About Content
Image credit: DARPA

51. Learning Registry
- Social Network sharing metadata and paradata of Learning Resources
Make contributing
as easy as possible !

52. Paradata
While learning analytics generally refers to analysis of data about learners,
paradata refers to data about learning resources.
Paradata can also record contextual information by linking resources with
educational standards and curricula, pedagogic approaches and methodologies.
Paradata can be regarded as an extended and altered version of JSON
ActivityStreams.
Paradata differs from ActivityStreams in that it enables complex aggregations of
activities to be recorded; e.g. High school English teachers taught using this
resource 15 times during the month of May 2011.

53. Content Strategy
xAPI modernize the SCORM runtime, what about a modernized content strategy?
IEEE LTSC : the answer is HTML5 + EPUB 3
W3C standards define an Open Web Platform for application development that has
the unprecedented potential to enable developers to build rich interactive
experiences, powered by vast data stores, that are available on any device.
=> HTML5 is the cornerstone.
In addition to the classic “Web of documents” W3C is helping to build a technology stack to support a “Web of data,” the sort
of data you find in databases. The ultimate goal of the Web of data is to enable computers to do more useful work and to
develop systems that can support trusted interactions over the network. The term “Semantic Web” refers to W3C’s vision
of the Web of linked data. Semantic Web technologies enable people to create data stores on the Web, build vocabularies,
and write rules for handling data. Linked data are empowered by technologies such as RDF, SPARQL, OWL, and SKOS.

54. RUSSEL - Re-Usability Support System for eLearning
The RUSSEL project includes:
● a novel user interface (UI),
● drag-and-drop SCORM disaggregation,
● a framework for automated metadata generation,
● a new approach to supporting instructional design best
practices while remixing existing materials,
● paradata capture from use of content in instructional
designs.

55. The Whole Picture

56. xAPI and Open Ecosystem Picture
- Based on standardized tracking and communicating language - xAPI, and the
standardization for learner's data
- Modularized ecosystem, new apps could always integrate in and with each other
(through standard web service call), a product could have several modular
components shown here
Knowledge Map / Topic graph / Competency
network
Learning Objects / Resources / Tools / Widgets
Adaptive engine / Intervention engine /
Recommendation algorithm
Analytics / Data mining tools
User Interface Dashboard / Reporting / Visualization
Learning Community Certifications

57. Next Gen Learning Environment
from ADL Andy Johnson

58. Building “Community of Practice(COP)” is the crucial
foundation of a standard, and the culture of using data for
making instructional decisions.
xAPI is an open and flexible framework standard for
tracking learning experiences, it needs further
“community-driven” rules - top-down rules won’t work -
for different domains to
ensure data quality and interoperability.