Organizing to Ret Analytics RightVince Kellen, Ph.D. Senior Vice Provost, Analytics and TechnologiesVince.Kellen@uky.eduThis is a living document subject to substantial revision! September, 2014

Silos
Are recursive
Get reproduced across time
and space reliably, without
effort
Arise naturally due to human
sociological/biological
tendencies
It takes constant effort to
mitigate their adverse effects
Sharing data and analysis
widely requires a
reconceptualization of silo
structures
2

Organization dysfunction
Information as power
Defensiveness
Data hoarding
Process separation
Empire building
Excessive control
Fear of scrutiny
Loss of power
3

We are competitive animals
Information becomes a [tool, weapon]
We instinctually manage information to enhance our competitiveness
Competition relies on information hiding
IT tools become part of our body
How we personally utilize information is part of our biological heritage. This is hardto change, if at all
4

Shift from production concerns to consumption ones

Production
•
Collecting, integrating, cataloging, categorizing, transforming, abstracting, analyzing, model-building, visualization, dashboarding, distributing, publishing. If you build it they will come (hopefully)

Consumption
•
Motivating, collaborating, expressing, integrating, improving action, increasing ambition, desire, recognition. If theybuild it everyonewill come
5

A Proposed Analytic Maturity Scale
6

Our process
7

A.
Merging of mobile and BI strategy
B.
Merging of IR and BI units
C.
Super high-speed infrastructure
D.
Single analytic value chain
E.
Analytics community of practice
F.
Data transparency
G.
Community sourcing and norming
H.
Community rules of etiquette
8

Our Community of Practice Rules of Etiquette
Be safe and secure. Respect the acceptable use of information policies and guidelines the university has in place. Please have
good passwords and secure your laptop, desktop and other devices appropriately. Treat private student and university
information appropriately.
Be collegial. University data is a community asset and a community of people steward the data. Use and share the data with
the best interests of the university community in mind. Since parts of our data analysis environment is designed to allow for
greater transparency, analysis will potentially be able to see other unit data. While we will make private to a unit what absolutely
needs to be private, the way the university runs it's business often involves multiple colleges and units at the same time. Don't
use your access to take unfair advantage of another unit.
Help improve data quality. If you see data that doesn't appear to be correct, let someone know. We have a team of staff
dedicated to helping improve data quality. This team can work with colleges and units on any data entry and data management
processes that might need to be changed to improve data quality.
Be open-minded and inquisitive. Data can be represented in multiple ways at the same time. While the teams are taking great
care to enable multiple views of the data to support the community, you might have a valid and unique perspective. In time, we
can accommodate more ways of looking at the same data while not interfering with other views or taxonomies.
Share. The main benefit from open analytics is the power of a community of analysts learning from each other rather than a few
select individuals hoarding knowledge or access. As the community improves its knowledge and skill with the data, the university
can improve accordingly.
9

Organizing IT

Our organizational model makes a big difference. Other universities fail to take advantage of a tool like this for purely political reasons

Making key data transparent to all does not help those who made their living being the data ‘go to’ person

We had to merge two units (Institutional Research and Business Intelligence), losing 1/3 of the staff. This let us hire three data scientists with different analytic backgrounds

The tool let the staff transition their skills
10

What we have done and what we would like to do

First steps over the past year
•
Mobile micro-surveys: Learning from the learner. In one year, 134,458 surveys harvested. Survey response rates are holding at about 40%. We can instantly analyze all responses for retention and progression issues
•
Student enrollment, retention, demographics, performance, K-Score, facilities utilization, instructor workload, student revenue and financial aid, student progression and more
•
High speed, in-memory analytics architecture. Lowest level of detail, maximum semantic expressiveness, one- second per click for analyst are key design philosophies
•
Open data and organizational considerations

Coming down the road?
•
Micro-segmentation tool to enhance user and IT productivity, develop personalized mobile student interaction/intervention
•
Models for learner technographics, psychographics, in addition to behaviors, performance, background
•
Advanced way-finding for streaming content like lecture capture
•
Content metadata extraction and learner knowledge discovery
•
Real-time measures of concept engagement and mastery
•
Real-time learner recommendations and support engine
•
Use graphing algorithms to perform more sophisticated degree audit what ifs
11

Our analytics technology infrastructure roadmap 2014-15
12

List builder

Iteratively query any/all fields of your choosing, linking in an AND or ORfashion

Combine different lists using SET manipulations

Refresh lists regularly (nightly or otherwise)

Apply the set name as a filter on ALL models

This provides advanced filtering and combining that works regardless of the user interface

Our AA team can build and maintain Lists easily. So can some users

Since lists are refreshed nightly, we can keep track of each time a student (or other entity) as it added or removed from a list

We can develop workflow apps using this. Backend, front-end agnostic
13

List builder
example
14

List builder
example
15

2009201020112012201320142015Academic year81012141618202224262830323436 Avg 19109107810716016111836154551551654716214195110916216556Fast/Slow ProgressionStudent headcount850100165Cohort YearFall 2008Fall 2009Fall 2010Fall 2011Fall 2012
List builder visualization example
Found all students who take a lot of classes at one point in their career and then took less classes at another point in their career.
Interpretation: These students start with a bang but fade at the finish
How long did this analysis take? Start to finish with this visualization:
25 minutes
16

K-Feed: Intelligent, personalized alerts, news, reminders
17

Identifying smaller segments of students
In addition to our work on difficult student cases, we needed to find a way to reach a ‘murky middle’ group of students
Identify students who are just as likely to come back as they are not
The predicted reenrollment was about 50%
After interventions, the actual enrollment was about 65%

The whole enchilada
Personalize learning, learning analytics and IPAS analytics into one real-time architecture
•
Real-time personalized interactions
•
Target on-demand peer tutoring based on student’s profile
•
Deliver micro-surveys and assessments to capture additional information needed to improve personalization
•
Give students academic health indicators that tell students where they can improve in study, engagement, support, etc.
•
Let students opt their parents in to this information so the family can support the student
•
Tailor and target reminder services, avoid over messaging, enable timing of message delivery based on user temporal proclivities, mix and match messages across learning, support and progression areas
•
Allow for open personalized learning
•
How content gets matched to students is psychologically complex
•
Several theories of how humans learn give many insights
•
Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning, cognitive reflection, need for sensation, need for cognition, various verbal abilities, confidence, persistence, prospective memory, etc.
•
We need an open architecture to promote rapid experimentation, testing and sharing of what works and what doesn’t
University of Kentucky

Herding cats

We shared with everyone that we are building the bridge as we walked on it

We established a community of practice and rules of analysis etiquette

We built tailored objects for colleges, let users choose their own front end tool

We relied on word-of-mouth adoption and some teasing-revealing

Guess what happened?
20

Top-down versus bottom-up

Doing this top down is like pushing water uphill. Its harder than pushing a rock uphill

The great leader is one who the people say “We did this ourselves”

Consider analytics to be a process of self discovery. Each person has to go through the stages of maturity

Paradoxically, this also requires strong top-down commitment and action! Organizational maneuvers like reorganizations are [normally] required
21

Questions?
22