Watch the video of this presentation: https://youtu.be/Qc9T5mj3oDc?t=1s Data mining is a multi-industry trend that looks certain to grow in both scope and application. It offers a hugely powerful means of identifying aggregates and trends at all levels, which is why we’ve worked extensively on deepening our analytics APIs over the last year. This session will walk you through the latest advancements to our analytics and show you how we can make your data more valuable and how we handle the heavy lifting to make it easier for you to create complex reports at individual, class, school, or district level.

1. Mining for Gold:
Using reporting and analytics to get
more value from your data
2:20 – 3:00pm
Michael & Denis

2. Michael Sharman
Director of Product & Engineering
michael.sharman@learnosity.com
Denis Hoctor
Director of Product & Business Intelligence
denis.hoctor@learnosity.com
Presenters

3. 1. What sort of data we collect
2. Reporting
○ For students
○ For teachers
3. Extending reports
4. Analysis
○ Aggregate reports
Outline

4. ● Make your data more valuable for you
● Make complex reporting easy
● Provide analysis tools
● Handle the heavy lifting
Goals

5. Data collection

6. Data collection
Data warehouse

7. Data collection
Responses
● Question data
● Responses
● Attempted
Session
● No personally identifiable
information
● Number of questions / attempted
● Status (completed or not)
● Date started / completed
● Activity id
● Session id
● User id

8. Data collection
Metadata
● Time per item
● Student flagged
● User agent
● Bank / pool source
● Custom
Scores
● Session, question and item scores
● Auto scored data
● Manually scored data
● Subscore data

9. Data collection → reporting
Data warehouse Reporting
Sessions
Responses
Scores
Metadata

10. ● Series of widgets customers can combine in any way
○ Student
○ Teacher
○ District administrators
● Configurable
● Extendable
Reports API

11. Reporting
Student facing

12. Reporting - student facing
Session summary
by question
Session summary
by tagCalculated via
Reports API

13. Reporting - student facing
Session detail

14. Reporting - student facing Last score by
activity
Last score by
item

15. Reporting
Teacher facing

16. Reporting - teacher facing
Session list
Last score by activity

17. Reporting - teacher facing
Last score by item
Last score by tag

18. Reporting
Teacher facing – Real time status

19. ● See student progress in real time
● Control students individually
○ Pause
○ Extend time
○ Save or discard
○ Handle navigation
Reports API

20. Reporting - proctor real time view

21. Extending reporting
Reports and Data APIs

22. Default report - Reports API
// Data available in Reports API `lastscore-by-activity`
{
"count_items": 15,
"count_items_attempted": 12,
"count_items_correct": 7,
"count_items_incorrect": 5,
"count_items_not_attempted": 3,
"max_score": "13.00",
"max_score_of_attempted": "12.00",
"not_scored": "0.00",
"percentage_items_attempted": 80,
"percentage_items_correct": 47,
"percentage_items_incorrect": 33,
"percentage_items_not_attempted": 20,
"score": "7.00",
"session_id": "76b681eb-1495-491f-bf05-c07af82ef",
"time_completed": "2017-04-22 08:21:01",
"time_spent_correct": 141,
"time_spent_incorrect": 36,
"time_spent_total": 236,
"time_spent_unattempted": 59,
"time_spent_unmarked": null
}
Last score by activity

23. Custom report - Reports API
// Data available in Reports API `lastscore-by-activity`
{
"count_items": 15,
"count_items_attempted": 12,
"count_items_correct": 7,
"count_items_incorrect": 5,
"count_items_not_attempted": 3,
"max_score": "13.00",
"max_score_of_attempted": "12.00",
"not_scored": "0.00",
"percentage_items_attempted": 80,
"percentage_items_correct": 47,
"percentage_items_incorrect": 33,
"percentage_items_not_attempted": 20,
"score": "7.00",
"session_id": "76b681eb-1495-491f-bf05-c07a182ef",
"time_completed": "2017-04-22 08:21:01",
"time_spent_correct": 141,
"time_spent_incorrect": 36,
"time_spent_total": 236,
"time_spent_unattempted": 59,
"time_spent_unmarked": null
}
Custom visualization for a class

24. Custom report - Reports API
// Data available in Reports API `lastscore-single`
{
"count_items": 15,
"count_items_attempted": 12,
"time_completed": "2017-04-22 08:21:01",
"score": "7.00",
"not_scored": "0.00",
"max_score": "13.00",
"max_score_of_attempted": "12.00",
"time_spent_correct": 141,
"time_spent_incorrect": 36,
"time_spent_unattempted": 59,
"time_spent_unmarked": null,
"time_spent_total": 236,
"count_items_correct": 7,
"session_id": "76b681eb-1495-491f-bf05-c07a16cf82ef"
}
Disable rendering

25. Custom report - Data API
Data API
Client
Raw data

26. Custom report - Data API
// https://data.learnosity.com/v1/sessions/responses
{
"user_id": "3848674363",
"activity_id": "learnosity-internal-demo",
"num_attempted": 13,
"num_questions": 15,
"session_id": "76b681eb-1495-491f-bf05-c07af82ef",
"score": 8,
"max_score": 26,
"subscores": [...],
"session_duration": 646,
"status": "Completed",
"dt_started": "2017-04-22T07:15:00Z",
"dt_completed": "2017-04-22T08:21:01Z",
"responses": [...],
"items": [...],
"metadata": {...}
}
Percent and score by subskill
Setup subscore groups before
students attempt a session!

27. Custom report - Data API
// https://data.learnosity.com/v1/sessions/statuses
{
"user_id": "67876543",
"activity_id": "natural_gases_grade_4_6",
"num_attempted": 5,
"num_questions": 9,
"session_id": "d9e87ada-60a0-4ee2-9d09-502c45c91df3",
"session_duration": 1,
"status": "Completed",
"dt_started": "2017-04-24T04:49:00Z",
"dt_completed": "2017-04-24T04:49:56Z"
},
{
"user_id": "1456754",
"activity_id": "natural_gases_grade_4_6",
"num_attempted": 4,
"num_questions": 9,
"session_id": "b3aaa172-aeee-487f-aa82-71191b54a232",
"session_duration": 1,
"status": "Incomplete",
"dt_started": "2017-04-24T04:48:00Z",
"dt_completed": null
}, ...
Status report

28. Reporting at scale
Aggregate (cohort)

29. Data collection → reporting
Data warehouse Reporting
Sessions
Responses
Scores
Metadata

30. Data collection → analysis → reporting
Data warehouse ReportingAnalysis Engine

31. Use cases
● Policy makers, district administrators analysing education data
● Principals or teachers looking to understand class performance
● Anyone authorized to see statistical analysis across a cohort
Aggregate reports

32. Aggregate reports
● Visually summarize, explore and compare the assessment
results for large cohorts of students (up to 200,000)
● Compare mean, median and other statistical measures
● Precomputed, point-in-time snapshot of scores
● Arbitrary groupings of users that you define
● Browse and navigate through the group hierarchy

33. Northern District
District B
Aggregate Report
(group hierarchy)
District C

34. Northern District
Springfield High
School B
School C
Schools
District C
Schools
District B
Aggregate Report
(group hierarchy)

35. Northern District
Springfield High
School B
School C
Class B
Class C
Schools
Classes
District C
Schools
Classes
District B
Aggregate Report
(group hierarchy)
Ms Hoover

36. Northern District
Springfield High
Schools
Classes
District B
Ms Hoover
Aggregate Report
(group hierarchy)
Compare:
● Mean
● Median
● Min/max score
● nth percentile
● And lots more...

37. Springfield High
Female
Sex
Attendance
School B
Attendance: 90%+
Male
80-89%
70-79%
Compare:
● Mean
● Median
● Min/max score
● nth percentile
● And lots more...
Aggregate Report
(group hierarchy)
Intersex

38. Aggregate reports
● Principle of Least Data
● No PII required
● Anonymized data stored
● Linked by host system
{
district_1: "Phoenix"
district_2: "Scottsdale"
school_1: "Scottsdale High"
school_2: "Phoenix Middle School"
school_3: "Phoenix High"
}
Customer Records:
{
student_id: "2017-322-4432",
uniq_ref: "12345678",
first_name: "Gavin",
last_name: "Cooney",
state: "AZ",
school: "Phoenix High",
district: "Phoenix",
race: "Caucasian",
socio_e_g:: "middle class",
free_lunch: "no",
res_type: "suburban",
...
}
Separation
of Data
Learnosity records
{
uniq_ref: 12345678
tests_completed: 14
questions_attempted: 186
...
}
Learnosity:
{
uniq_ref: "12345678"
hierarchy_1: "district_1"
hierarchy_2: "school_3"
hierarchy_3: "r_1"
hierarchy_4: "seg_2"
percentile_75: "0.84"
percentile_50: "0.64"
percentile_25: "0.45"
}

39. Demo:
Aggregate report
https://docs.staging.learnosity.com/demos/isolation/reports/activitysummarybygroup/ui-demo.php

40. Aggregate reports
● State
● District
● School
● Class
● Cultural factors
● Rural suburban urban
● Socio economic
● Attendance
● Dyslexic
● School type
● Special needs program
● Family factors
Example groups / hierarchies

41. Data
API
Aggregate reports - Generation
Analysis Engine
Reports
API
Anonymized groups
Requested fields
Raw data

42. Aggregate reports - Summary
● Completely customizable
● Access to all the raw data
● Processed asynchronously (precomputed)
● Preserves your privacy and security
● Extensible in powerful ways

43. Extending aggregate reports
Longitudinal report based off multiple aggregate reports
Semester 1 Semester 2 Semester 3 Semester 4
Report ReportReportReport

44. Analytics
Looking ahead

45. Analytics
Item bank analysis
● Break down of banks
○ Question types
○ Configurations (single vs multi select MC)
○ Items by tag
■ Eg ELA vs Math
■ Standards
■ DOK

46. Analytics
Item score analysis
● Cohort / time based classical item analysis
● Help check item reliability and validity
○ Relationship of item scores to test scores
■ P-Value
■ Point biserial

47. Analytics - Item bank
Items with bad
validation
Lack of easy
items in this bank

48. Analytics - Item bank

49. Analysis - more to come
Data warehouse ReportingAnalysis Engine

50. Q&A

51. Mining for Gold:
Using reporting and analytics to get
more value from your data
2:20 – 3:00pm
Michael & Denis

52. Custom report - Reports API
// Data available in Reports API `lastscore-by-activity`
{
"count_items": 15,
"count_items_attempted": 12,
"count_items_correct": 7,
"count_items_incorrect": 5,
"count_items_not_attempted": 3,
"max_score": "13.00",
"max_score_of_attempted": "12.00",
"not_scored": "0.00",
"percentage_items_attempted": 80,
"percentage_items_correct": 47,
"percentage_items_incorrect": 33,
"percentage_items_not_attempted": 20,
"score": "7.00",
"session_id": "76b681eb-1495-491f-bf05-c07a182ef",
"time_completed": "2017-04-22 08:21:01",
"time_spent_correct": 141,
"time_spent_incorrect": 36,
"time_spent_total": 236,
"time_spent_unattempted": 59,
"time_spent_unmarked": null
}
Custom charting library