Confronting Data Manipulation

1. Confronting Data
Manipulation on Your
Campus
Dr. Will Miller
Director of Institutional Research and Effectiveness
Flagler College

3. Context
• We rely on others for data
• IPEDS
• Outcomes reports
• Closing the Loop
• Course evaluations
• Technology has eliminated the paper trail

4. Data Manipulation
• Comes in a variety of flavors
• Different than data error!
• Survey responses
• Course evaluations
• Course outcomes
• Closing the Loop reports
• Or more troubling forms

5. Flagler College
• Founded in 1968
• Baccalaureate only
• Approximately 2700 students
• U.S. News Top Regional
• The unicorn of admissions
• Increase applicant pool
• Increase cohort size
• Decrease merit scholarships
• Increase class profile

6. But…
• Unicorns aren’t real

7. The Swift Fall of the Unicorn
• Background of the offender
• New Director of IRE and New Director of College Writing in Summer
2013
• During a random audit of student placements from fall semester, we
notice a vast majority of students were under-placed in writing
courses according to information in Student Management System
• Initially blame advisors
• But, test scores in the system do not match hardcopy test scores in
student files
• The two cases that sealed his fate

8. Internal Process
• Notification of VPAA (February 7)
• More analysis (Weekend of February 8-9)
• Audit of data entry changes shows one user affiliated with all changes on weekends in
September for numerous years (beginning in 2004)
• Update of VPAA (February 9 around 2am)
• Notification of President (February 10 at 8am)
• More analysis
• Final notification of President (February 12 at 8am)
• Trip with VPEM (February 13)
• Removal (February 14)
• Measure Impact (Weekend of February 15-16)
• Notify Cabinet on February 17 at 8am

9. Sampling of Findings

10. Sampling of Findings

11. Contributing Factors
• Relationships
• Whistle blower
• Access to databases
• Test score entry
• Policies on paper v. policies in practice
• Silos

12. The Public Process
• External investigators
• External data auditors
• Crisis communication consultants
• Openness and transparency
• Stakeholders
• Campus community

13. Sampling of Findings

14. Sampling of Findings

15. Reporting and Response
• Open and transparent
• Easier to tell all at the beginning
• No sugar coating
• Responses from community
• Responses from stakeholders
• Accrediting bodies
• Government
• Publications

16. Impact
• Faculty
• Students
• Staff
• Institution as a whole

17. And the Other Shoe Falls
• Students aren’t impacted…if they take math and English first
semester
• Which some students don’t
• 223 students end up misplaced in math, English, or in a few
cases both due to test score fraud
• A whole new process
• Pull transcripts, courses taken, time of registration v. time of data
manipulation, determine remediation while consulting with faculty,
presenting remediation, and in some cases scrub transcripts
• Interests DoE

18. Lessons Learned
Any press does not necessarily make for good press
Be open and transparent
Be selective in choosing investigators
Do not worry about cost…you’ll be remembered by how
you respond

19. Lessons Learned
Do not sugarcoat
Policies exist to be followed
Do not hold press conferences
Some people don’t need access to some things

20. Lessons Learned
Do not blindly trust

21. Lessons Learned
It can be a positive impetus for change
• We’re actually a unicorn
• Decreased applications, increased yield, made the class (and then
some), and increased profile
• Re-established relationships between enrollment management and
faculty
• Personnel changes

22. Questions