Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data-Driven Learning Strategy


Published on

Education must capitalize on the trend within technology toward big data. New types of data are becoming available. From evidence approaches to xAPI and the whole Training and Learning Architecture(TLA) big data is the foundation of all.

Published in: Education
  • Be the first to comment

Data-Driven Learning Strategy

  1. 1. Data-Driven Learning Strategy Jessie Chuang Classroom Aid Inc.
  2. 2. Data is the only hard evidence of what works in education, and the compass for personalized learning journey.
  3. 3. Data, Data, Data Amazon: data driven engine accounts for 1/3 of all sales Netflix: recommendation engine accounts for 3/4 of all new orders How about learning and training segment?
  4. 4. Evidence Framework A report by U.S. Dep. of Ed. - discusses the promise of sophisticated digital learning systems for collecting and analyzing very large amounts of fine-grained data (“big data”) as users interact with the systems. - describes an iterative R&D process, with rapid design cycles and built-in feedback loops—one familiar in industry but less so in education (however, the report provides numerous examples of applications in education).
  5. 5. Two critical steps - 1 ● Education must capitalize on the trend within technology toward big data. ● New types of data are becoming available. ● broad, static categories(demographic, end-of-year grades and test scores) => more dynamic, fine-grained data in the process of learning ● data in a single context => linked across different parts of learning ● collected through different methodologies and reported in isolation => aggregating evidences from different data sources
  6. 6. Two critical steps - 2 ● A revitalized framework for analyzing and using evidence ○ Gold standard evidence - slow and expensive, poor matches to the rapid pace of digital development practices ○ An evidence framework should help educational stakeholders align their methods of obtaining evidence with their goals, the risks involved, the level of confidence needed, and the resources available. Confidence Risk The need of evidence before implementation collection of data should be an ongoing task
  7. 7. Challenges addressed by evidence approaches ● Making sure learning resources promote deeper learning ● Building adaptive learning systems that supports personalized learning ● Combining data to create support systems more responsive to student needs ● Improving the content and process of assessment with technology ● Finding appropriate learning resources and making informed choices
  8. 8. Evidence Approaches w/i Great Potential 1. Educational data mining and learning analytics applied to data gathered from digital learning systems implemented at scale 2. Rapid A/B testing conducted with large numbers of users within digital learning systems (example: Khan Academy) 3. Design-based implementation research (DBIR) supported by data gathered from digital learning systems (collaborative design from developers, researcher, teachers) 4. Large datasets of different types from multiple sources, combined and shared across projects and organizations 5. Technology-supported evidence-centered design (ECD) of measures of student learning 6. Data gathered from users about a learning resource, how users have used it and their experiences using it. => Example: enable “Rapid Prototyping” by making early version of a product freely available online, collect data and mine data for insights; ratings and reviews from users …...
  9. 9. What Works Clearinghouse (by U.S. Department of Education) - “The high volume of research on different programs, products, practices, and policies in education can make it difficult to interpret and apply the results. We review the research. Then, by focusing on the results from high-quality research, we try to answer the question “What works in education?” - The goal is to provide educators with the information they need to make evidence-based decisions.
  10. 10. iZone - Researchers and Schools collaborate on an Education Test Bed ● Partnership from 250 schools, Research Alliance (based at NYU), EdSurge, ChallengePost, and IDEO ● Supporting from U.S. Dep. of Education, Bill&Melinda Gates Foundation, New York City Dep. of Education ● To support developers in rapidly developing and testing selected technology based instructional supports and featuring test results on EdSurge ● To address unmet needs identified by a diverse group of school stakeholders ● Launch prize competitions for developers, winners will be invited to beta-test products in iZone classrooms ● Researcher have access to resources necessary to a given study; test bed data are available for secondary analysis to continue relevant study (also to develop analytic capability and let good algorithms bubble up) ● The collaboration will produce Consumer Reports-style guide for learning tech. ● Silicon Valley also initiated an iZone to create a similar ecosystem
  11. 11.
  12. 12. Learning Analytics Applications ● Learner modeling: model a learner’s knowledge, behaviors, motivation, experience… ● Learner profiling: cluster users into similar groups ● Domain modeling: decompose content into components and sequences ● Effectiveness: test learning principles, pedagogies etc. ● Trend analysis: understand changes over time ● Recommendation and adaptation: suggest resources and actions to learners or instructors, adapt systems to learners...
  13. 13. Research examples Combine observation data with click streams to detect boredom, frustration, gaming the system… ● Off-task behaviors patterns can be identified like completing module too fast or too slow … ● Evidences of gaming the systems like systematic and rapid incorrect answers, systematic guessing … Experience API, xAPI, can give us more because …
  14. 14. Learner *Motivator *Navigator (content brokering) *e-Portfolio(or digital badge backpack) inform iterate co-design Data Analyst Learning Designer Standards Interoperability Analysis (data mining) Learning theories Psychology Design (experience design, UI) Subject Matter Expertise integrate (developers are constructors) *Connecting Learner Community *Learner Attribute (style) *Narrator (learner created context, reflection) Gamified layer on data from across platforms Big Data [Data-Driven Learning Strategy]
  15. 15. Intelligent learning environment is to match a learner with the right content at the right time.
  16. 16. About Learner Profiling Image credit: DARPA
  17. 17. Recording Learning Events Learning happens in interactions: Contents: Courses, Books, Web pages, Games, AR …. Instructors, Peers, Experts…. Activities(making, exercises, researching, online, offline ….) Learner
  18. 18. Recording Learning Events Social Learning Group Learning image credit: Search Engine People Blog
  19. 19. Learning is different
  20. 20. image credit: “A New Architecture for Learning”, published by Educause
  21. 21. xAPI tracking all kinds of learning experiences
  22. 22. The xAPI specification has two primary parts 1. defines the syntax of the xAPI data format a. the vocabularies should be community-driven b. all activities and context can be tracked c. any enabled application/device can send statements 2. defines the characteristics of “learning record stores” (LRS) - a crucial component of xAPI a. data can be exchanged between LRSs (set free from LMS) b. learner can have life-long “personal learning locker” c. LRSs need to validate xAPI statements
  23. 23. Social Coaching Project Learning Mobile Apps LRS Learning data is sent to LRS Other activities Course Webpage Game Simulator
  24. 24. LMS LRS LRS Reporting Tool Learning records can be delivered to LMSs,LRSs or Reporting Tools.
  25. 25. Recording Learning Events ActivityStreams: stream of activity data statements, borrowed from social analytics This is only the basic idea. Crafting the statements with more context and related information is necessary to support analytics and reporting.
  26. 26. 11 Attributes in xAPI Data Format ★ Unique Identifier ★ Actor (required) ★ Verb (required) ★ Object (required) ★ Result ★ Context ★ Timestamp ★ Stored (internal recording timestamp) ★ Authority ★ (Protocol) Version ★ Attachments All information in XAPI statements can be separated into : ● meta-data, ● descriptive information, and ● complementary data.
  27. 27. Syntax-1 Actor: Agent (= persona) or group (multiple IDs allowed) Verb: ID = an IRI(URL) = a specific semantic meaning + human readable display Object: an agent, a group, a statement or an activity(most common)
  28. 28. Syntax-2 Activity: ID = an IRI(URL) = with a specific boundary (granularity) Definition: Name Description objectType Extentions (useful to customize reporting)
  29. 29. Syntax-3 Result: Score Success Duration Completion Response (learner’s response to the experience) Extentions
  30. 30. Syntax-4 Context: Registration (differentiate multiple attempts) Instructor Team ContextActivities (parent, grouping, category, other - like related lesson) Revision Platform Language Statement (refer to one other statement for a whole experience) Extentions
  31. 31. Context, Context, Context The XAPI differentiates between the core context and the wider context. The core context includes the instructor(s), the direct peers involved in an activity (team), the learning environment (platform), the language that was used in the performance, and a framing statement for an activity (e.g., the course that relates to the activity). The extended context includes a set of data-records about the wider context of a learning activity. This wider context is not explicitly specified and can include the location of the learner, the wider (social) relations, the duration of an activity, environmental factors (e.g., temperature or noise level) etc. The format and the content of the wider context is specific to the AP and is not subject to the interoperability of the data format.
  32. 32. Ask the right questions!
  33. 33. LRS An LRS is defined by two interfaces: ● Statement interface (statement API) ● Document interface - this interface handles three types of documents ● State interface (state API) ● Activity profile interface (activity API) ● Agent profile interface (agent API) The LRS is responsible for 1. validating that the system sending data is authorized, 2. checking that the data being sent is xAPI-compliant, 3. storing the data properly, 4. making that data available to any other authorized system or activity provider when asked.
  34. 34. Advanced Applications of xAPI - Data Transfer based on RESTful HTTP w/i LRS ● Agent Profile API ○ personal info., learner profile and modeling, user settings, learning journal, career plan & goal ○ an integrated picture of a learner activities across systems and devices with multiple identities ● Activity Profile API (for activity provider) ○ interactions between learners (collaboration, social or competition) ○ learning planning tool (access to or update the LRS internal definition of a given activity id, even before the activity sends any statement) ● State API ○ persist state across devices ● Authentication services, querying services, visualization services, and personal data services are some examples.
  35. 35. xAPI tracking is... Semantic Contextualized From any device and sensor LRS frees the learning data so they can be put together, analyzed, modeled, reused, carried with learners and accumulated life-long
  36. 36. xAPI + Open Badges => Learner ePortfolio Both are representing learner data by exploiting HTTP, JSON, and REST - simple, lightweight method that lowers entry barrier for developers. Together, they offer a new way to think about constructing interoperable learner model data!
  37. 37. JSON vs XML ● JSON is bandwidth-non-intensive ● JSON is better adapted (than XML) to devices with limited capabilities such as smart things ● JSON possesses a very limited set of data types. Restricting itself to primitive data types makes it deeply and immediately interoperable with pretty much any programming language that exists out there. ● JSON is a better data exchange format, XML is a better document exchange format ● JSON is a preferred format in NoSQL database - data not seamlessly conform to a columnar/relational model. Since JSON objects may be heterogeneous in terms of number and types of fields, this allows for tremendous flexibility in storing and retrieving objects as compared to relational databases. (Should you like to know more about JSON, here is a very simple interactive tutorial. )
  38. 38. Bootstrapping Learner Model Data ● Making data contribution as easy as possible ● Bootstrapping heterogeneous learner model data ○ the raw experience data stored in a LRS referenced by an Open Badge, could be analyzed to perform an application’s own interpretations of that evidence, as suggested by Carmagnola, Cena, and Gena (2011) ○ Guo & Greer, 2007; ○ Tiroshi, Kuflik, Kay & Kummerfeld, 2011 ) summarized various boostrapping methods ○ ADL developed an open source project/resource called “lr-data” (Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)
  39. 39. Sharing and Analysis of Data In the case of sharing and making sense of learner data, a similar model can be followed where any contributing application (e.g., an ITS or game) could easily publish activity data to a LRS, Open Badges to a learner’s badge backpack, or other learner model data to their shared learner profile as JSON over a RESTful HTTP connection. The LRS, badge backpack, or learner profile can exist anywhere (e.g., an employer organization, a commercial provider, or even self-hosted by a learner) – all the contributing applications need to know are the URLs and the learners credentials. If a learning application wants to make sense of all this data, it too just needs the URLs and the learner’s credentials to get started. The learner could set permissions concerning what applications can access their data. (Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)
  40. 40. TLA Services TLA will also include services for managing learner profiles. Open Badges can be referenced by learner profiles, which will likely contain other learner data such as goals, reflection, etc. The TLA will also include services for creating and accessing competency definitions to serve as a common way to reference educational standards, learning objectives, and competency definitions through web APIs (Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)
  41. 41. The whole picture = Training and Learning Architecture(TLA) ● ePortfolio ● Learner modeling ● Machine readable ● Competency standards ● Knowledge map ● Standard alignment ● xAPI COP ● Common vocabulary ● Learning Design ● Sharing of metadata & paradata (LR) ● Re-usability ● Semantic analysis
  42. 42. A taxonomy of education standards Redd, Brandt
  43. 43. Education Standards for Interoperability ❏ Academic standards ❏ Data Standards (consistent definition) ❏ Student data ❏ Data Dictionary (data element definition) ❏ Logical Data Model (entity definition) ❏ Educator data ❏ Content data (metadata, paradata) ❏ xAPI vocabulary/activity profile CEDS
  44. 44. Major Learner Data Categories ● Educational records ● Competencies(skills, knowledge, abilities, outcomes…) and domain learning objectives ● Data in affective, motivational(disposition) and social dimensions ● Data for learning style modeling ● Data for pedagogical purposes
  45. 45. About Content Image credit: DARPA
  46. 46. Learning Registry - Social Network sharing metadata and paradata of Learning Resources Make contributing as easy as possible !
  47. 47. Paradata While learning analytics generally refers to analysis of data about learners, paradata refers to data about learning resources. Paradata can also record contextual information by linking resources with educational standards and curricula, pedagogic approaches and methodologies. Paradata can be regarded as an extended and altered version of JSON ActivityStreams. Paradata differs from ActivityStreams in that it enables complex aggregations of activities to be recorded; e.g. High school English teachers taught using this resource 15 times during the month of May 2011.
  48. 48. Content Strategy xAPI modernize the SCORM runtime, what about a modernized content strategy? IEEE LTSC : the answer is HTML5 + EPUB 3 W3C standards define an Open Web Platform for application development that has the unprecedented potential to enable developers to build rich interactive experiences, powered by vast data stores, that are available on any device. => HTML5 is the cornerstone. In addition to the classic “Web of documents” W3C is helping to build a technology stack to support a “Web of data,” the sort of data you find in databases. The ultimate goal of the Web of data is to enable computers to do more useful work and to develop systems that can support trusted interactions over the network. The term “Semantic Web” refers to W3C’s vision of the Web of linked data. Semantic Web technologies enable people to create data stores on the Web, build vocabularies, and write rules for handling data. Linked data are empowered by technologies such as RDF, SPARQL, OWL, and SKOS.
  49. 49. RUSSEL - Re-Usability Support System for eLearning The RUSSEL project includes: ● a novel user interface (UI), ● drag-and-drop SCORM disaggregation, ● a framework for automated metadata generation, ● a new approach to supporting instructional design best practices while remixing existing materials, ● paradata capture from use of content in instructional designs.
  50. 50. The Whole Picture
  51. 51. xAPI and Open Ecosystem Picture - Based on standardized tracking and communicating language - xAPI, and the standardization for learner's data - Modularized ecosystem, new apps could always integrate in and with each other (through standard web service call), a product could have several modular components shown here Knowledge Map / Topic graph / Competency network Learning Objects / Resources / Tools / Widgets Adaptive engine / Intervention engine / Recommendation algorithm Analytics / Data mining tools User Interface Dashboard / Reporting / Visualization Learning Community Certifications
  52. 52. Next Gen Learning Environment from ADL Andy Johnson
  53. 53. Building “Community of Practice(COP)” is the crucial foundation of a standard, and the culture of using data for making instructional decisions. xAPI is an open and flexible framework standard for tracking learning experiences, it needs further “community-driven” rules - top-down rules won’t work - for different domains to ensure data quality and interoperability.