Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Rethinking Student Success 
Analytics in Support of Teaching and Learning 
Timothy D. Harfield, Scholar in Residence (Lear...
2 
Why Learning Analytics? 
@tdharfield
 Founded in 1836 
 11 Schools 
(4 Undergraduate, 7 Graduate) 
 14,513 Students 
(7,836 Undergraduate, 6,677 Graduate & ...
• “The most dramatic factor shaping the future of higher education” 
• Drop in U.S. News & World Report Rankings, from 20t...
5 
Blackboard Analytics Pilot 
@tdharfield
@tdhar6field 
Blackboard Analytics Pilot 
Methodology 
Blackboard Analytics for Learn™ Adds to Emory’s existing analytical...
Overall 
Ease of Use 
@tdhar7field 
Blackboard Analytics Pilot 
Integrated Reports 
Would You Like to Have Access 
Yes 
51...
@tdhar8field 
Blackboard Analytics Pilot 
Integrated Reports | Student Feedback 
“I believe it's important for professors ...
@tdhar9field 
Blackboard Analytics Pilot 
Integrated Reports | Faculty Feedback 
• Anecdotally, faculty report that studen...
@tdha1r0field 
Blackboard Analytics Pilot 
Online Analytical Processing 
• Delivered reports for the Pyramid BI reporting ...
11 
62% 
38% 
Course Items Added No Course Items Added 
DFW Rate by Blackboard Use 
5% 5% 
100% 
75% 
50% 
25% 
0% 
Course...
@tdha1r2field 
Blackboard Analytics Pilot 
Exploratory Data Mining 
DFW Rate by Design Quartile 
4.8% 
5.2% 
5.8% 
4.0% 
5...
@tdha1r3field 
Blackboard Analytics Pilot 
Exploratory Data Mining 
DFW Rate by Class 
3.3% 
4.1% 
7.3% 
5.5% 
10% 
5% 
0%...
DFW Rate by Class 
(Oxford College) 
10.1% 9.8% 
Junior Senior 
@tdha1r4field 
Blackboard Analytics Pilot 
Exploratory Dat...
15 
5.3% 
6.1% 
4.8% 4.8% 
10% 
5% 
0% 
Files Items 
DFW Rate by Item Type 
DFW Rate by Link Type 
5.3% 5.4% 
1.0% 
4.0% 
...
16 
DFW Rate by Grade Center Use 
4.1% 
5.8% 
10% 
5% 
0% 
No Yes 
@tdharfield 
Blackboard Analytics Pilot 
Exploratory Da...
17 
DFW Rate by Minutes Z-Score 
2.9% 
4.3% 
5.8% 6.5% 
20% 
10% 
0% 
>= 1 >= 0 & < 1 > -1 & > 0 <= -1 
DFW Rate by Intera...
18 
Next Steps 
@tdharfield
@tdha1r9field 
Regional Community of Practice 
“Data Wrangling” 
Local Community of Practice 
Next Steps
@tdha2r0field 
Next Steps 
Data Wrangling 
Predicting Students at Risk 
 Data Validation 
 Data Dictionary 
 Predictive...
@tdha2r1field 
Next Steps 
Data Wrangling 
Online Course Evaluation Framework 
Roxanne Russell, PhD 
 Are logged course b...
@tdha2r2field 
Next Steps 
Data Wrangling 
Getting a Leg Up at Emory (GLUE) 
Drew Kohlhorst, PhD 
 Does participation in ...
@tdha2r3field 
Next Steps 
Data Wrangling 
Emory Coursera Initiative 
Stephanie Parisi 
 Who are our learners? 
 Why do ...
@tdha2r4field 
Next Steps 
Local Community of Practice 
Analytics for Learning at Emory (ALE) 
 Receive inspiration and a...
@tdha2r5field 
Next Steps 
Regional Community of Practice 
https://scholarblogs.emory.edu/ale
THANKS! 
Timothy D. Harfield 
Scholar in Residence (Learning Analytics) 
timothy.harfield@emory.edu 
@tdharfield
Upcoming SlideShare
Loading in …5
×

Rethinking Student Success: Analytics in Support of Teaching and Learning

1,346 views

Published on

Presented at the 2014 Blackboard Institutional Performance Conference (30-31 October 2014).

ABSTRACT: Passing grades and retention through to degree are essential to success in higher education, but these factors are too often mistaken for ends in themselves. A high-performing student environment has provided teachers and researchers at Emory University with a space to think critically about what success means, and about the extent to which data might inform the design of successful learning environments. This presentation will (1) discuss some of the unique challenges encountered by Emory University during its 2013-2014 Blackboard Analytics pilot, (2) describe several provisional insights gained from exploratory data mining, and (3) outline how Emory’s pilot experience has informed support of learning analytics on campus. What we have learned at Emory University has both broad and deep implications for how institutions use data in support of student success, but these insights could only have been achieved in an environment where grade-performance and retention are not significant issues.

Published in: Education
  • I'd advise you to use this service: ⇒ www.WritePaper.info ⇐ The price of your order will depend on the deadline and type of paper (e.g. bachelor, undergraduate etc). The more time you have before the deadline - the less price of the order you will have. Thus, this service offers high-quality essays at the optimal price.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Rethinking Student Success: Analytics in Support of Teaching and Learning

  1. 1. Rethinking Student Success Analytics in Support of Teaching and Learning Timothy D. Harfield, Scholar in Residence (Learning Analytics) Emory University Libraries and Information Technology Services @tdharfield
  2. 2. 2 Why Learning Analytics? @tdharfield
  3. 3.  Founded in 1836  11 Schools (4 Undergraduate, 7 Graduate)  14,513 Students (7,836 Undergraduate, 6,677 Graduate & Professional)  Average Freshman Retention Rate: 95.3%  4-year Graduation Rate: 84%  94% College Success Rate (<> DFW)  US News National University Ranking #21 Fact & Figures as of Fall 2013 (http://www.emory.edu/home/about/factsfigures/) @tdharfield Why Learning Analytics? Emory University
  4. 4. • “The most dramatic factor shaping the future of higher education” • Drop in U.S. News & World Report Rankings, from 20th to 21st • Existing rates of student success makes at-risk students difficult to identify Why Blackboard Analytics for Learn? • Learning Management Gap in Oracle BI Solution • Fast, lightweight, inexpensive @tdhar4field Why Learning Analytics? Emory University
  5. 5. 5 Blackboard Analytics Pilot @tdharfield
  6. 6. @tdhar6field Blackboard Analytics Pilot Methodology Blackboard Analytics for Learn™ Adds to Emory’s existing analytical capacity In three areas: I. Learning Analytics Integrated Blackboard Reports for teachers and students  Survey Instrument based on Learning Analytics Acceptance Model II. Academic Analytics Online Analytical Processing for instructional designers and university leadership  Focus Group III. Educational Data Mining / Predictive Analytics Direct database queries and table exports for longitudinal modeling ( >180 days) and analysis using statistical software packages and machine learning algorithms  Exploratory Data Mining
  7. 7. Overall Ease of Use @tdhar7field Blackboard Analytics Pilot Integrated Reports Would You Like to Have Access Yes 51% in Other Courses? No 9% Maybe 40% 6%3% 27% 21% 43% 3% 23% 47% 15% 12% Overall Usefulness Would You Recommend It Yes 36% to Others? No 17% Maybe 47% Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree
  8. 8. @tdhar8field Blackboard Analytics Pilot Integrated Reports | Student Feedback “I believe it's important for professors to have this tool available to them, so they can see early if we're not turning up for class enough and intervention might be warranted. I do know that I need to get in a habit of logging onto BB at least twice a day. I need to make it part of my daily life” “I liked being able to compare my activity to others, but I don't think it changed my behavior. It may be useful to others.” “I think its useful if someone is concerned about how they are doing in the class or if they are uncertain about how much time they should be committing to the class. I personally did not find it useful but I can see how someone may use it as a tool to determine if they are meeting expectations” “I think this system also puts undue pressure on students to log in or click on specific items in blackboard more, without taking into consideration factors such as work done offline in the class or via email. This is especially true for distance learners. I personally cut and paste many of my assignments in Word so they are easier to view and manipulate. I also like to keep files from each semester so I do a considerable amount of work offline and I am doing very well in my classes so far. When I logged in to check my activity, the data I saw pretty much showed the opposite-it looked like I wasn't very engaged at all.” “If this is a tool that will be used by Professors to grade participation I would have likely used it more; however, if this tool were indeed used by Professors it may cause students to post unnecessarily.” “You should be a big button at the front of the page with an icon to remind people to use it.”
  9. 9. @tdhar9field Blackboard Analytics Pilot Integrated Reports | Faculty Feedback • Anecdotally, faculty report that students like the new analytics feature, although it is a distraction for some high performing students who become overly driven by metrics at the expense of actual course objectives Takeaway: Use of Course Analytics should be accompanied by clear, continuous communication about the tool’s use and limitations, and the ways in which data are being used for assessment. There is much to be gained from the use of embedded analytics, but also risk if not implemented in manner that is pedagogically sound.
  10. 10. @tdha1r0field Blackboard Analytics Pilot Online Analytical Processing • Delivered reports for the Pyramid BI reporting environment are of little value to instructional designers. a. Data dictionary is out of date, inaccurate, confusing, and inconsistent b. It is unclear what various reports are meant to accomplish, and how they might help generate actionable insights Takeaways: Reporting using the Pyramid BI reporting environment is more useful to the Office of Institutional Research than to instructional designers Instructional designers are interested in analyzing data using more inferential and predictive techniques. Program improvement should start with questions rather than with data.
  11. 11. 11 62% 38% Course Items Added No Course Items Added DFW Rate by Blackboard Use 5% 5% 100% 75% 50% 25% 0% Course Items Added No Course Items Added Course Items Added No Course Items Added @tdharfield Blackboard Analytics Pilot Exploratory Data Mining
  12. 12. @tdha1r2field Blackboard Analytics Pilot Exploratory Data Mining DFW Rate by Design Quartile 4.8% 5.2% 5.8% 4.0% 5.1% 10% 5% 0% First Quartile Second Quartile Third Quartile Fourth Quartile Non-Blackboard
  13. 13. @tdha1r3field Blackboard Analytics Pilot Exploratory Data Mining DFW Rate by Class 3.3% 4.1% 7.3% 5.5% 10% 5% 0% Freshman Sophomore Junior Senior
  14. 14. DFW Rate by Class (Oxford College) 10.1% 9.8% Junior Senior @tdha1r4field Blackboard Analytics Pilot Exploratory Data Mining DFW Rate by Class (Excluding Oxford College) 3.3% 4.1% 20% 10% 0% 6.4% 4.4% 10% 5% 0% Freshman Sophomore Junior Senior
  15. 15. 15 5.3% 6.1% 4.8% 4.8% 10% 5% 0% Files Items DFW Rate by Item Type DFW Rate by Link Type 5.3% 5.4% 1.0% 4.0% 10% 5% 0% Course Links External URLs Not Used Used @tdharfield Blackboard Analytics Pilot Exploratory Data Mining
  16. 16. 16 DFW Rate by Grade Center Use 4.1% 5.8% 10% 5% 0% No Yes @tdharfield Blackboard Analytics Pilot Exploratory Data Mining
  17. 17. 17 DFW Rate by Minutes Z-Score 2.9% 4.3% 5.8% 6.5% 20% 10% 0% >= 1 >= 0 & < 1 > -1 & > 0 <= -1 DFW Rate by Interactions Z-Score 2.8% 3.5% 6.5% 8.6% 20% 10% 0% >= 1 >= 0 & < 1 > -1 & > 0 <= -1 DFW Rate by Access Z-Score 2.5% 3.4% 6.7% 10.3% 20% 10% 0% >= 1 >= 0 & < 1 > -1 & > 0 <= -1 @tdharfield Blackboard Analytics Pilot Exploratory Data Mining
  18. 18. 18 Next Steps @tdharfield
  19. 19. @tdha1r9field Regional Community of Practice “Data Wrangling” Local Community of Practice Next Steps
  20. 20. @tdha2r0field Next Steps Data Wrangling Predicting Students at Risk  Data Validation  Data Dictionary  Predictive Modeling
  21. 21. @tdha2r1field Next Steps Data Wrangling Online Course Evaluation Framework Roxanne Russell, PhD  Are logged course behaviors predictive of student satisfaction with course design, implementation, and interaction dynamics?  Can we develop algorithms for identifying pain points, and effective interventions to mitigate dissatisfaction and other barriers to learning?
  22. 22. @tdha2r2field Next Steps Data Wrangling Getting a Leg Up at Emory (GLUE) Drew Kohlhorst, PhD  Does participation in the GLUE program contribute significantly to student success in STEM courses at Emory?  Does GLUE function to attract and retain more STEM majors?  Are there significant differences in the success rates of students participating in the online versus residential versions of the program?  Does participation see changes in attitudes/confidence in scientific skills, and/or changes in on-campus resource utilization?
  23. 23. @tdha2r3field Next Steps Data Wrangling Emory Coursera Initiative Stephanie Parisi  Who are our learners?  Why do learners take a particular course?  What factors contribute most to learner satisfaction?
  24. 24. @tdha2r4field Next Steps Local Community of Practice Analytics for Learning at Emory (ALE)  Receive inspiration and assistance in the formation of questions that might guide the use of data to optimize learning and learning environments  Share current interests and projects, in order to receive feedback and encouragement, and to develop possibilities for future collaboration  Gain exposure to the field of learning analytics (language, key concepts, methodologies, etc) through access to experts
  25. 25. @tdha2r5field Next Steps Regional Community of Practice https://scholarblogs.emory.edu/ale
  26. 26. THANKS! Timothy D. Harfield Scholar in Residence (Learning Analytics) timothy.harfield@emory.edu @tdharfield

×