Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Learning	Analytics:	Harnessing	Data	
Science	to	Transform	Education
Tim	McKay:	University	of	Michigan
@TimMcKayUM,	Blog	at...
The	University	of	Michigan
• 200	yr old	public	research	intensive	university	with	
19	Schools	and	Colleges	&	6,800	faculty...
Particle	astrophysics:
Carefully	measuring	nothing…
Observational	Cosmology:
Measuring	Everything…
This	is	observational	s...
Teaching	thousands	of	students	
introductory	physics	of	every	kind…
8	years	as	Director	of	LSA	Honors	
Program,	caring	for...
The	20th Century	began	with	an	industrial	revolution.
Public	higher	education	joined	in:	exploding	in	scale	and	
adopted	b...
The	21st Century	began	with	an	information	revolution.
We	know	more	about	our	students	than	we	ever	have	and	connect	them	...
Remember	the	goal…
1984
Why	hasn’t	this	happened	already?
Why	so	much	visceral	resistance	to	
the	idea	of	using	technology	to	
personalize	educati...
• >98%	of	these	are:	
• Small	&	changeable
• Taught	in	idiosyncratic,	
engaged,	and	creative	ways
• <2%	of	them	are:
• Lar...
Why	personalize?	
For	the	735	students	in	my	class…
Nita
Frank
We	are	often	fooled	by	a	false	
sense	of	personalization.	T...
Focus	on	Education	@	Scale
• Even	at	big	Universities,	most	courses	have	
what	they	need	to	be	excellent.	Large	
introduct...
Five	themes	of	our	work
Learning	analytics	for	personalization
1. Ethics:	what	are	we	doing	and	why
2. Measurement:	data	c...
#1	Ethics:
What	we’re	doing	and	why
Information	ethics:	a	grand	challenge
• What	principles	should	govern	collection	and	
use	of	data	about	individuals	in	edu...
Six	Asilomar principles
1. Respect	for	the	rights	and	dignity	of	learners:	transparency,	
consent,	protection	of	privacy
2...
Example	1:	Predictive	Modeling
As	educators,	we	learn	from	the	past	
in	order	to	change the	future…
Example	2:	Probing	Inequity
Too	often,	we	assume	that	identical	
treatment	models	fairness…
Example	3:	Measureable	Types
• Data	often	used	to	
categorize,	collecting	
individuals	into	groups
• These	labels	are	ofte...
Intersectionality	in	LA
• Learn	methodology	of	intersectionality	from	
feminist	scholarship*:	use	multiple	approaches
– In...
#2	Measurement:	
Data	collection	and	management
What	do	we	measure?
• What	we	measure	now:
– Admissions	information
– Course	taking	&	grades
– Degrees	&	honors
• What	we’...
Just	for	student	records,	
there	are	157	pages	of	data	
description…hundreds	of	
organically	evolving,	
interacting	tables...
Example	partial	solution:	UM	Learning	
Analytics	Data	Architecture
A	‘regular	release’	model	for	clean	
research	data.	Sim...
How	to	release	information while	
protecting	privacy?
• As	much	as	possible,	we	should	let	everyone	
learn	from	the	experi...
Better	measures	of	learning
• Grades:	performance	
measures	of	unrecorded	
tasks,	meant	to	estimate	
unknown	outcomes,	
qu...
Intellectual	Breadth Disciplinary	Depth Range	of	Experience
Engagement	&	Effort Social	&	Professional
Networks
Academic	
P...
Students	connect	through	courses
Courses	connect	through	students
Can	we	quantify	intellectual	breadth?
Explore	each	stude...
Connected	in	
major
Connected	
out	of	
major
#3	Analysis:	
Learning	from	experience
Methods	for	reliable	inference	from	
observational	data
Three	examples	using	different	
methodologies:
1. Are	our	classrooms	equitable?
2. Do	learning	communities	work?
3. Are	pl...
BTE
WTE
#1:	Koester/Grom/McKay
Are	our	classrooms	equitable?	
Student	performance	is	influenced	by	
background	and	prepara...
Gendered	performance	
differences
<GPA	– Grade>	Male	=	0.32
<GPA	– Grade>	Female	=	0.59
GPD	=	0.27
These	performance	
diff...
All	large	intro	STEM	
lecture	courses	+	
Econ	101/102
Measured	of	grade	penalty	&	GPD	across	all	large	courses	at	UM:	
Str...
Data	from	2000	– 2012	for	all	large		STEM	lecture	and	lab	courses
Lab	courses
Lecture	courses
Details,	including	tests	of	...
Biology Chemistry
Math	&	Stats Physics
Data	from	five	Big	10	Schools:	
Similar	GPD	patterns	across	
lecture	&	lab	STEM	cou...
#2:	Brooks/Morgan/Maltby - HSSP	Impact
Living	
Learning	
Quasiexperimental design
Health	Science	Scholars	Program
Example	results
HSSP	significantly	increased	the	likelihood	of	BS	
and	advanced	degrees	for	underrepresented	and	
first-ge...
Michigan	1:2:1	Introductory	Chemistry	Curriculum	Model:
Traditional	2:2	Introductory	Chemistry	Curriculum	Model:
2 Semeste...
How	does	taking	Gen	Chem first	matter?
Chem
Placement
Shultz,	Ginger	V.,	Amy	C.	Gottfried,	and	Grace	A.	Winschel.	 Journal...
What	can	go	wrong	with	all	
of	these	methods?	
Education	is	harder	than	physics…
Replicability	≠ generalizability
Action:	Putting	data	to	work
Decision	making,	story	telling,	
motivating	change
How	to	put	data	to	work…
A	spectrum	of	information	agency…
Give	students,	
advisors,	faculty	
the	data	~directly.	
Let	the...
UM	Digital	Innovation	
Greenhouse
Take	good	ideas	developed	
on	campus	from	innovation	
to	infrastructure,	support	
Ed-Tec...
University	
Teaching	
Community
UNIZIN
Startups
External	
Research	
Funding
Research	
Findings	&	
Pubs
Innovators	&	
pione...
DIG	was	born	in	May	2015:	
A	place,	a	team	of	innovators,	
originally	in	DEI	Lab	on	Washington	
DIG	has	grown,	
and	now	li...
DIG	team	connects	faculty/staff
FACULTY	DIRECTOR
Tim	McKay
OPERATIONS	DIRECTOR
Mike	Daniel
FACULTY	CHAMPIONS
Gus	Evrard	(L...
Students	are	our	best	
creative	engine:	Fellows,
Design	Jams	&	Hackathons!
DIG	projects:	
A	rapidly	growing	portfolio
And	more….
Providing	information	to	
individuals
Learning	about	classes	and	more:	
ART	2.0
ART	2.0	– information	to	all
Course	cards	will	be	joined	by	reports	on	
courses	of	study	(majors	and	minors)	
and	people	(...
Expert	interpretation	and	advice	
at	scale
ECoach:	computer	tailored	electronic	
coaching	for	equity	and	student	success
Expert	tailored	communication
• Built	on	20+	yrs of	digital	health	coaching
• Aggregates	rich	student	info	from	many	sourc...
ECoach:	
richly	tailored	
messaging,	
informed	by	
behavioral	
science	and	
disciplinary	
expertise
ECoach	is	expanding	to	more	
classes,	launching	at	other	
institutions,	and	supporting	
rich	array	of	research	
projects	w...
This	intervention	launched	this	
fall	as	an	RCT	with	more	than	
1000	students	in	each	of	the	
treatment	and	control	arms.
...
Synthesis:
Creating	a	learning	laboratory	for	
studying	education	at	scale
Focus	on	Foundational	Courses
• Large,	relatively	stable,	
mostly	introductory	courses
• Serve	students	with	
especially	v...
A	Learning	Higher	Ed	System
Digital	Innovation	Greenhouse
Large	course	team	develops	and	
supports	a	technical	infrastruct...
Five	years	from	now…
• Carefully	designed	and	instrumented	
foundational	courses	established	as	one	of	the	
key	elements	o...
Educating	@	scale	in	21st century	
• Teaching	at	scale	in	the	
information	age	affords	
unprecedented	
opportunities	for	
...
Upcoming SlideShare
Loading in …5
×

Keynote – Timothy McKay – Learning Analytics: Harnessing Data Science to Transform Education – OWD17

968 views

Published on

The 21st century opened amidst an information revolution which promises to transform higher education as dramatically in this century as industrialization did in the last. Many things have already changed, but the real revolution will come when we harness information technology to personalize education: optimizing our education of an increasingly diverse student body, creating much greater student motivation and engagement, and accomplishing more with less.

This talk describes how research using data about the processes and products of education led the University of Michigan to discover patterns of inequity in STEM education, develop an array of new student support technologies, and launch a major new Foundational Course Initiative.

Published in: Education
  • Fact: Penis Enlargement CAN Work. Here's How. ★★★ http://t.cn/Ai88iYkP
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • If you just broke up with your Ex, you have to follow these steps to get him back or risk ruining your chances. ▲▲▲ http://ow.ly/f23I301xGAo
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Keynote – Timothy McKay – Learning Analytics: Harnessing Data Science to Transform Education – OWD17

  1. 1. Learning Analytics: Harnessing Data Science to Transform Education Tim McKay: University of Michigan @TimMcKayUM, Blog at 21stCenturyHigherEd.com
  2. 2. The University of Michigan • 200 yr old public research intensive university with 19 Schools and Colleges & 6,800 faculty • Highly selective group of 29,000 undergraduate and 16,000 graduate students • Annual budget of $7 billion, including $1.4 billion in federally funded research (#1 public)
  3. 3. Particle astrophysics: Carefully measuring nothing… Observational Cosmology: Measuring Everything… This is observational science: drawing inference without classical methods of experiment…
  4. 4. Teaching thousands of students introductory physics of every kind… 8 years as Director of LSA Honors Program, caring for 2000 students across the disciplines… My need for data about these students & their experiences led to continually expanding efforts in Learning Analytics
  5. 5. The 20th Century began with an industrial revolution. Public higher education joined in: exploding in scale and adopted bureaucratic, industrial approaches, including standardized tests, credit hours, GPAs, majors, and minors. Too often, we pursue a 20th century, industrial form of optimization: seeking a single system which maximizes learning across a population of students.
  6. 6. The 21st Century began with an information revolution. We know more about our students than we ever have and connect them with us, information, one another, and the world in unprecedented ways. Our goal today is a 21st century, information age form of optimization: adapting the system to individually optimize learning for each student.
  7. 7. Remember the goal… 1984
  8. 8. Why hasn’t this happened already? Why so much visceral resistance to the idea of using technology to personalize education?
  9. 9. • >98% of these are: • Small & changeable • Taught in idiosyncratic, engaged, and creative ways • <2% of them are: • Large & relatively stable • Taught in industrial, remote, and tradition bound ways One reason? There are two ways to teach a large number of students… We have 9200 courses at UM
  10. 10. Why personalize? For the 735 students in my class… Nita Frank We are often fooled by a false sense of personalization. These two students are 0.3% of the whole pool…
  11. 11. Focus on Education @ Scale • Even at big Universities, most courses have what they need to be excellent. Large introductory and many online courses don’t – They could be dramatically better with a different approach to instruction and professional support • Improving education at scale is a major sociotechnical challenge, requiring both: – New information technology for personalization – New social norms for course design & delivery It’s hard to beat Bloom’s tutor where she can act. Learning analytics should focus on improving things where she cannot.
  12. 12. Five themes of our work Learning analytics for personalization 1. Ethics: what are we doing and why 2. Measurement: data collection and management 3. Analysis: modeling, extraction of meaning, learning from the experience of all 4. Action: decision making, storytelling, creating the motivation for change 5. Synthesis: Building a learning laboratory
  13. 13. #1 Ethics: What we’re doing and why
  14. 14. Information ethics: a grand challenge • What principles should govern collection and use of data about individuals in education? – What data is relevant for education? – Norms of consent, privacy, autonomy? How are they different within an academic community? – How are experiments in educational practice related to research norms? • We need to ensure that commercial Ed-Tech is a part of (& constrained by) this conversation
  15. 15. Six Asilomar principles 1. Respect for the rights and dignity of learners: transparency, consent, protection of privacy 2. Beneficence: maximize benefits, minimize harm 3. Justice: benefit all, reduce inequalities 4. Openness: learning and research are public goods 5. The humanity of learning: insight, judgment, & discretion are essential, we should keep learning humane 6. Continuous consideration: ongoing, inclusive discussion of changing ethical circumstances http://asilomar-highered.info/ Where to have this conversation?
  16. 16. Example 1: Predictive Modeling As educators, we learn from the past in order to change the future…
  17. 17. Example 2: Probing Inequity Too often, we assume that identical treatment models fairness…
  18. 18. Example 3: Measureable Types • Data often used to categorize, collecting individuals into groups • These labels are often reductive and invisible to learners • Always incomplete, substituting a category for individuals • Cheney-Lippold (2017) ‘measureable types’ ≠ complex socially constructed classes – gender ≠ ‘gender’ – text ≠ ‘positive’ • Categorizations too often one dimensional, excluding intersections of identity How to limit the impact of reflexively reductive data representation?
  19. 19. Intersectionality in LA • Learn methodology of intersectionality from feminist scholarship*: use multiple approaches – Intercategorical: focus on variation across socially constructed provisional categories – Intracategorical: analyze variation w/in categories – Anticategorical: real personalization, no categories • Be transparent, allow for agency: resist labelling individuals w/o understanding and consent *McCall, L. (2005). The complexity of intersectionality. Signs: Journal of women in culture and society, 30(3), 1771-1800.
  20. 20. #2 Measurement: Data collection and management
  21. 21. What do we measure? • What we measure now: – Admissions information – Course taking & grades – Degrees & honors • What we’re starting to record (explosive growth) – Process of learning: clickstreams, discussions, video, course structures – Products of learning: forum posts, essays, papers, presentations, theses • What we want to have: Detailed, relevant, evolving portraits of every student's background, interests, goals, and accomplishments • These portraits should be used to help students, faculty, administrators, staff better understand higher education
  22. 22. Just for student records, there are 157 pages of data description…hundreds of organically evolving, interacting tables… 1st challenge: Data cleaning & aggregation
  23. 23. Example partial solution: UM Learning Analytics Data Architecture A ‘regular release’ model for clean research data. Similar to those in open science projects like the SDSS or GAIA space mission
  24. 24. How to release information while protecting privacy? • As much as possible, we should let everyone learn from the experience of all – Restricted reporting tools – access to digested information, within tools (Ex: ART 2.0, ECoach) – Existing research protocols – IRB oversight, anonymization => the LARC approach • New approaches are emerging in data science: synthetic data contain all the information but none of the details • Personal privacy can (& must) be protected well. We should rethink institutional privacy…
  25. 25. Better measures of learning • Grades: performance measures of unrecorded tasks, meant to estimate unknown outcomes, quantified on ill-defined scales • We should be measuring learning – increases in well defined knowledge and skills – and focusing on individual growth over time • Direct: pre and post testing aligned with learning goals. Good for foundational courses? • Indirect: Data Science tools for extracting meaning from products – Simple: IRT, topic modeling and beyond – Complex: peer evaluation, NLP, direct representation rather than data reduction
  26. 26. Intellectual Breadth Disciplinary Depth Range of Experience Engagement & Effort Social & Professional Networks Academic Performance Measuring what matters: the Transcript of the Future
  27. 27. Students connect through courses Courses connect through students Can we quantify intellectual breadth? Explore each student’s network of connection • Course co-enrollment: well measured, large bipartite network • Better representations of interaction coming Compare measured network structures to appropriate random graphs – measure diversity of connection Exposes isolation of majors, allows comparison of individuals within a major
  28. 28. Connected in major Connected out of major
  29. 29. #3 Analysis: Learning from experience Methods for reliable inference from observational data
  30. 30. Three examples using different methodologies: 1. Are our classrooms equitable? 2. Do learning communities work? 3. Are placement exams used well?
  31. 31. BTE WTE #1: Koester/Grom/McKay Are our classrooms equitable? Student performance is influenced by background and preparation. For example: grades in physics related to grades in other courses. Observed correlation Classroom equity
  32. 32. Gendered performance differences <GPA – Grade> Male = 0.32 <GPA – Grade> Female = 0.59 GPD = 0.27 These performance differences remain when we account for all measures of background & preparation. Unexplained performance differences like this are signs of classroom inequity. We must look for and address these disparate impacts. Koester, Grom, McKay: https://arxiv.org/abs/1608.07565 Classroom equity
  33. 33. All large intro STEM lecture courses + Econ 101/102 Measured of grade penalty & GPD across all large courses at UM: Striking patterns of gendered performance difference Classroom equity
  34. 34. Data from 2000 – 2012 for all large STEM lecture and lab courses Lab courses Lecture courses Details, including tests of many other possible performance predictors: arXiv 1608:07565 Intercategorical complexity Classroom equity
  35. 35. Biology Chemistry Math & Stats Physics Data from five Big 10 Schools: Similar GPD patterns across lecture & lab STEM courses. Matz et al. AERA Open in press This analysis uses both hierarchical linear modeling and quasi-experimental matching methods
  36. 36. #2: Brooks/Morgan/Maltby - HSSP Impact Living Learning Quasiexperimental design Health Science Scholars Program
  37. 37. Example results HSSP significantly increased the likelihood of BS and advanced degrees for underrepresented and first-generation students. Living Learning Quasiexperimental design
  38. 38. Michigan 1:2:1 Introductory Chemistry Curriculum Model: Traditional 2:2 Introductory Chemistry Curriculum Model: 2 Semesters General Chemistry 2 Semesters Organic Chemistry Chemistry 130 Chemistry 210 & 215 Chemistry 230 #3: Shultz/Gottfried/Winschel Chemistry Placement Analysis Chem Placement Regression discontinuity
  39. 39. How does taking Gen Chem first matter? Chem Placement Shultz, Ginger V., Amy C. Gottfried, and Grace A. Winschel. Journal of Chemical Education 92.9 (2015): 1449-1455. Regression discontinuity
  40. 40. What can go wrong with all of these methods? Education is harder than physics… Replicability ≠ generalizability
  41. 41. Action: Putting data to work Decision making, story telling, motivating change
  42. 42. How to put data to work… A spectrum of information agency… Give students, advisors, faculty the data ~directly. Let them decide what to do. Give ‘experts’ the data. Have them interpret, and make decisions for students, advisors, faculty. Give the data to both! Have experts help students, advisors, faculty interpret data: shape decisions using behavioral science, choice architecture, nudges Information & advising systems always face a spectrum of agency. What’s new is the richness of information and analysis. To do this, you need tools which protect privacy while sharing information, professionally designed for their users.
  43. 43. UM Digital Innovation Greenhouse Take good ideas developed on campus from innovation to infrastructure, support Ed-Tech R&D, personalize education at scale
  44. 44. University Teaching Community UNIZIN Startups External Research Funding Research Findings & Pubs Innovators & pioneering adopters DIG team of Developers, U/X Designers, Behavioral Scientists Communities of practice: faculty, students, staff University Research Community University IT: support at scale DIG: a home for academic R&D
  45. 45. DIG was born in May 2015: A place, a team of innovators, originally in DEI Lab on Washington DIG has grown, and now lives atop our library
  46. 46. DIG team connects faculty/staff FACULTY DIRECTOR Tim McKay OPERATIONS DIRECTOR Mike Daniel FACULTY CHAMPIONS Gus Evrard (LSA) Barry Fishman (SI) Elisabeth Gerber (Ford) Anne Gere (Sweetland) Tim McKay (LSA) Perry Samson (ENG) Ginger Schultz (LSA) LEAD BEHAVIORAL SCIENTIST Holly Derry LEAD DEVELOPERS Ben Hayward Cait Holman Kris Steinhoff Chris Teplovs LEAD INNOVATION ADVOCATE Amy Homkes-Hayes BEHAVIOR SCIENTIST Carly Thanhouser DATA SCIENTIST Kyle Schulz DEVELOPERS Dave Harlan Kushank Raghav Oliver Saunders Ke Ye UX & DESIGN Marie Hooper Kristin Miller Mike Wojan Plus 15-20 student fellows drawn from Computer Science, Social Psychology, Art & Design, UI Design, Behavioral Science, Education, & more….
  47. 47. Students are our best creative engine: Fellows, Design Jams & Hackathons!
  48. 48. DIG projects: A rapidly growing portfolio And more….
  49. 49. Providing information to individuals Learning about classes and more: ART 2.0
  50. 50. ART 2.0 – information to all Course cards will be joined by reports on courses of study (majors and minors) and people (students, faculty), along with tools for curriculum exploration…
  51. 51. Expert interpretation and advice at scale ECoach: computer tailored electronic coaching for equity and student success
  52. 52. Expert tailored communication • Built on 20+ yrs of digital health coaching • Aggregates rich student info from many sources to tailor feedback, encouragement, & advice • Tailoring on both what to say, how to say it, who speaks: w/testimonials from peers, etc. • All content written & tested by behavior change experts, faculty from disciplines, students • A tool for humane personalization w/student agency: allowing us to speak, share data, connect
  53. 53. ECoach: richly tailored messaging, informed by behavioral science and disciplinary expertise
  54. 54. ECoach is expanding to more classes, launching at other institutions, and supporting rich array of research projects w/external funding. This fall: 8000 students Stats 250 EECS 183, 280 Chem 130 Physics 140 Bio 171 Econ 101 Engr 100, 101 ALA 125 FirstYear (6800 more…) UC Santa Barbara ECoach Future ECoach supports research addressing widespread gendered performance differences in STEM lecture courses
  55. 55. This intervention launched this fall as an RCT with more than 1000 students in each of the treatment and control arms. First of a series of upcoming experiments delivered in ECoach Learning to respond to student writing using NLP etc. is a major goal for the coming year.
  56. 56. Synthesis: Creating a learning laboratory for studying education at scale
  57. 57. Focus on Foundational Courses • Large, relatively stable, mostly introductory courses • Serve students with especially various backgrounds • Serve students with especially various interests and goals • Foundational courses where we educate at scale are ideal environments for the application of analytics • Best large courses taught in multigenerational teams with role specialization • Roles include: – Course management – Delivery of instruction on large and small scales – Instructional design – Technology – Assessment & analytics – Student support • Courses should be broadly ‘instrumented’ for study These ‘foundational’ courses exist across many disciplines, most of which are outside the natural sciences, so this initiative is campus-wide.
  58. 58. A Learning Higher Ed System Digital Innovation Greenhouse Large course team develops and supports a technical infrastructure for research – gathering data and implementing intervention studies Foundational Course Initiative CCD process provides a social infrastructure for sustained research and development – practice generating research Data and Technology People & Social Systems Translational Research on Education at Scale: The FCI and DIG are creating the sociotechnical framework needed to support rich translational education research. We are bringing research teams into this Learning Laboratory, intimately connecting research & practice in the authentic, evolving environment of our foundational courses.
  59. 59. Five years from now… • Carefully designed and instrumented foundational courses established as one of the key elements of a learning laboratory • A well established program, with ~20-30 courses established as part of this laboratory • We aim to play an important role in establishing a robust evidence-basis for learning analytics & personalization at scale
  60. 60. Educating @ scale in 21st century • Teaching at scale in the information age affords unprecedented opportunities for personalization • Realizing these is a major sociotechnical challenge • We are addressing both the social and technical challenges associated with this task at Michigan • Our campus is creating a laboratory for learning at scale, connecting education research & practice • This will become a “learning higher education” community in which we learn continuously from experience in context • It can all be done in a student-centered way while protecting privacy

×