Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Organizing to Get Analytics Right

Deriving value from analytics requires much more than purchasing technology. University of Kentucky's analytics journey utilized fostering a bottom-up emergent community of practice as well as top-down organizational maneuvers. This presentation shares different aspects of the University of Kentucky score.

  • Login to see the comments

Organizing to Get Analytics Right

  1. 1. Organizing to Ret Analytics RightVince Kellen, Ph.D. Senior Vice Provost, Analytics and TechnologiesVince.Kellen@uky.eduThis is a living document subject to substantial revision! September, 2014
  2. 2. Silos Are recursive Get reproduced across time and space reliably, without effort Arise naturally due to human sociological/biological tendencies It takes constant effort to mitigate their adverse effects Sharing data and analysis widely requires a reconceptualization of silo structures 2
  3. 3. Organization dysfunction Information as power Defensiveness Data hoarding Process separation Empire building Excessive control Fear of scrutiny Loss of power 3
  4. 4. We are competitive animals Information becomes a [tool, weapon] We instinctually manage information to enhance our competitiveness Competition relies on information hiding IT tools become part of our body How we personally utilize information is part of our biological heritage. This is hardto change, if at all 4
  5. 5. Shift from production concerns to consumption ones  Production • Collecting, integrating, cataloging, categorizing, transforming, abstracting, analyzing, model-building, visualization, dashboarding, distributing, publishing. If you build it they will come (hopefully)  Consumption • Motivating, collaborating, expressing, integrating, improving action, increasing ambition, desire, recognition. If theybuild it everyonewill come 5
  6. 6. A Proposed Analytic Maturity Scale 6
  7. 7. Our process 7
  8. 8. A. Merging of mobile and BI strategy B. Merging of IR and BI units C. Super high-speed infrastructure D. Single analytic value chain E. Analytics community of practice F. Data transparency G. Community sourcing and norming H. Community rules of etiquette 8
  9. 9. Our Community of Practice Rules of Etiquette Be safe and secure. Respect the acceptable use of information policies and guidelines the university has in place. Please have good passwords and secure your laptop, desktop and other devices appropriately. Treat private student and university information appropriately. Be collegial. University data is a community asset and a community of people steward the data. Use and share the data with the best interests of the university community in mind. Since parts of our data analysis environment is designed to allow for greater transparency, analysis will potentially be able to see other unit data. While we will make private to a unit what absolutely needs to be private, the way the university runs it's business often involves multiple colleges and units at the same time. Don't use your access to take unfair advantage of another unit. Help improve data quality. If you see data that doesn't appear to be correct, let someone know. We have a team of staff dedicated to helping improve data quality. This team can work with colleges and units on any data entry and data management processes that might need to be changed to improve data quality. Be open-minded and inquisitive. Data can be represented in multiple ways at the same time. While the teams are taking great care to enable multiple views of the data to support the community, you might have a valid and unique perspective. In time, we can accommodate more ways of looking at the same data while not interfering with other views or taxonomies. Share. The main benefit from open analytics is the power of a community of analysts learning from each other rather than a few select individuals hoarding knowledge or access. As the community improves its knowledge and skill with the data, the university can improve accordingly. 9
  10. 10. Organizing IT  Our organizational model makes a big difference. Other universities fail to take advantage of a tool like this for purely political reasons  Making key data transparent to all does not help those who made their living being the data ‘go to’ person  We had to merge two units (Institutional Research and Business Intelligence), losing 1/3 of the staff. This let us hire three data scientists with different analytic backgrounds  The tool let the staff transition their skills 10
  11. 11. What we have done and what we would like to do  First steps over the past year • Mobile micro-surveys: Learning from the learner. In one year, 134,458 surveys harvested. Survey response rates are holding at about 40%. We can instantly analyze all responses for retention and progression issues • Student enrollment, retention, demographics, performance, K-Score, facilities utilization, instructor workload, student revenue and financial aid, student progression and more • High speed, in-memory analytics architecture. Lowest level of detail, maximum semantic expressiveness, one- second per click for analyst are key design philosophies • Open data and organizational considerations  Coming down the road? • Micro-segmentation tool to enhance user and IT productivity, develop personalized mobile student interaction/intervention • Models for learner technographics, psychographics, in addition to behaviors, performance, background • Advanced way-finding for streaming content like lecture capture • Content metadata extraction and learner knowledge discovery • Real-time measures of concept engagement and mastery • Real-time learner recommendations and support engine • Use graphing algorithms to perform more sophisticated degree audit what ifs 11
  12. 12. Our analytics technology infrastructure roadmap 2014-15 12
  13. 13. List builder  Iteratively query any/all fields of your choosing, linking in an AND or ORfashion  Combine different lists using SET manipulations  Refresh lists regularly (nightly or otherwise)  Apply the set name as a filter on ALL models  This provides advanced filtering and combining that works regardless of the user interface  Our AA team can build and maintain Lists easily. So can some users  Since lists are refreshed nightly, we can keep track of each time a student (or other entity) as it added or removed from a list  We can develop workflow apps using this. Backend, front-end agnostic 13
  14. 14. List builder example 14
  15. 15. List builder example 15
  16. 16. 2009201020112012201320142015Academic year81012141618202224262830323436 Avg 19109107810716016111836154551551654716214195110916216556Fast/Slow ProgressionStudent headcount850100165Cohort YearFall 2008Fall 2009Fall 2010Fall 2011Fall 2012 List builder visualization example Found all students who take a lot of classes at one point in their career and then took less classes at another point in their career. Interpretation: These students start with a bang but fade at the finish How long did this analysis take? Start to finish with this visualization: 25 minutes 16
  17. 17. K-Feed: Intelligent, personalized alerts, news, reminders 17
  18. 18. Identifying smaller segments of students In addition to our work on difficult student cases, we needed to find a way to reach a ‘murky middle’ group of students Identify students who are just as likely to come back as they are not The predicted reenrollment was about 50% After interventions, the actual enrollment was about 65%
  19. 19. The whole enchilada Personalize learning, learning analytics and IPAS analytics into one real-time architecture • Real-time personalized interactions • Target on-demand peer tutoring based on student’s profile • Deliver micro-surveys and assessments to capture additional information needed to improve personalization • Give students academic health indicators that tell students where they can improve in study, engagement, support, etc. • Let students opt their parents in to this information so the family can support the student • Tailor and target reminder services, avoid over messaging, enable timing of message delivery based on user temporal proclivities, mix and match messages across learning, support and progression areas • Allow for open personalized learning • How content gets matched to students is psychologically complex • Several theories of how humans learn give many insights • Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning, cognitive reflection, need for sensation, need for cognition, various verbal abilities, confidence, persistence, prospective memory, etc. • We need an open architecture to promote rapid experimentation, testing and sharing of what works and what doesn’t University of Kentucky
  20. 20. Herding cats  We shared with everyone that we are building the bridge as we walked on it  We established a community of practice and rules of analysis etiquette  We built tailored objects for colleges, let users choose their own front end tool  We relied on word-of-mouth adoption and some teasing-revealing  Guess what happened? 20
  21. 21. Top-down versus bottom-up  Doing this top down is like pushing water uphill. Its harder than pushing a rock uphill  The great leader is one who the people say “We did this ourselves”  Consider analytics to be a process of self discovery. Each person has to go through the stages of maturity  Paradoxically, this also requires strong top-down commitment and action! Organizational maneuvers like reorganizations are [normally] required 21
  22. 22. Questions? 22