1. Measuring What Matters
The Role of Non-Cognitive Factors
in Student Success
NUTN Webinar – February,
Dr. Mac Adkins, President, SmarterServices
Provided by

2. Question 1?
• How do you determine who can be
enrolled at your school?
– Standardized test scores
– Prior grade point averages
– Admissions exams

3. Top Admissions Factors
• The National Association for College Admission Counseling rated
these factors.
• CONSIDERABLY IMPORTANT
– College prep course grades
– Strength of high school curriculum
– Standardized test scores
– Overall GPA
• MODERATELY IMPORTANT
– Admissions essay
– Letters of recommendation
– Demonstrated interest
– Class rank
– Extracurricular commitment

4. Question 2
Why Do Students Drop Out?
A study funded by the Bill and Melinda Gates Foundation
ranked these reasons:
1. Conflict with work schedule
2. Affordability of tuition
3. Lack of support from family – financial and practical
support
4. Lack of belief that a college degree is valuable
5. Lack of discipline – too much socializing, not enough
studying
http://www.publicagenda.org/pages/with-their-whole-lives-ahead-of-them

6. To Find Out What Matters
Let’s Ask:
Employers
Colleges
Faculty
National Research Council
US Department of Education
Mothers

7. Skills that
Employer’s
Want
National Association of Colleges
and Employers
Survey of Employers
http://www.unl.edu/svcaa/documents/how_e
mployers_see_candidates.pdf

8. Outcomes Schools Want
Elements of Mission Statements From 35 Universities
1. Knowledge, learning, mastery of general principles
2. Continuous learning, intellectual interest, curiosity
3. Artistic cultural appreciation
4. Appreciation for diversity
5. Leadership
6. Interpersonal skills
7. Social responsibility, citizenship and involvement
8. Physical and psychosocial health
9. Career preparation
10.Adaptability and life skills
11.Perseverance
12.Ethics and integrity
Michigan State University, 2004

9. Traits Online Faculty Want
WICHE Cooperative for Educational Technologies, 2013

10. 2012 National Research Council
COGNITIVE
Problem solving
Critical thinking
Systems thinking
Study skills
Adaptability
Creativity
Meta-cognitive skills
INTERPERSONAL
Communication
Social Intelligence
Teamwork
Leadership
Cultural sensitivity
Tolerance for diversity
INTRAPERSONAL
Anxiety
Self-efficacy
Self-concept
Attributions
Work ethic
Persistence
Organization
Time management
Integrity
Life-long learning

11. US Department of Education
―The test score accountability movement and
conventional educational approaches tend to
focus on intellectual aspects of success, such as
content knowledge. However, this is not sufficient.
If students are to achieve their full potential, they
must have opportunities to engage and develop a
much richer set of skills. There is a growing
movement to explore the potential of the
―noncognitive‖ factors —
attributes, dispositions, social skills, attitudes, and
intrapersonal resources, independent of
intellectual ability—that high-achieving individuals
draw upon to accomplish success.‖

12. Parents Teach It

13. Are You Beginning To See The Picture?
• Non-cognitive skills matter
– Determine student retention
– Determine employer satisfaction
– Determine online course success
– Federal agencies recognize their importance
– They are the mission of many schools
– Parents value them

14. ―Years of schooling predicts labor market
outcomes — cognitive skills account for only
20%; therefore 80% of the ―years of
schooling‖ benefit is due to noncognitive
skills‖ (Bowles, Gintis, & Osborne, 2001)
http://www.umass.edu/preferen/gintis/jelpap.pdf

15. Types of Data Used To Predict
Learner Success
APTITUDE
ATTITUDE
SITUATION

16. What Are Non-Cognitive Skills?

17. Can Non-Cognitive Skills
Be Taught?
You can’t
change a tiger’s
stripes, but you
can teach that
tiger to hunt in a
different
environment.

18. Recommended Uses of
Non-Cognitive Skills Measures
1. Optic – A lens through which students can view their
strengths and opportunities for improvement
2. Student Service – A tool to guide students toward
available resources for support
3. Placement – Developmental / remedial course
placement
4. Talking Points – A collection of statements which
academic advisors can use to advise their students
5. Early Alert – A list of students who are likely to be
benefitted by the instructor reaching out to them early in
the course.
6. Predictive Analytic - A set of data which can be
analyzed at the individual and aggregate level to project
student performance

19. Methods of Measurement
•
•
•
•
•
Instructor ratings – Time and task intensive for the faculty
Observer records – Expensive and time consuming
Letters of recommendation – Rarely objective
Interviews – Time consuming to conduct and code
Socioeconomic data – Beneficial mostly at the aggregate level due
to exceptions and bias
• Self assessment – Yes, there are limitations, but it is the preferred
method.

20. Construct Comparison Matrix
ACT
Engage
Individual
Attributes
ETS Success
Navigator
Wonderlic
Admissions
Risk Profile
SmarterMeasure
X
X
X
X
Life Factors
X
Learning Styles
X
Technical Skills
X
X
Reading Skills
X
Keyboarding Skills
X
Custom Questions
X

21. SmarterMeasure
Learning Readiness Indicator
• A 124-item online skills test and attributes
inventory that measures a student’s level of
readiness for studying online
• Used by over 500 Colleges and Universities
• Since 2002 taken by over 2,600,000 students

22. What Does The Assessment Measure?
INTERNAL
EXTERNAL
SKILLS
INDIVIDUAL
ATTRIBUTES
LIFE FACTORS
TECHNICAL
Availability of Time
Dedicated Place
Reason
Support from Family
Technology Usage
Life Application
Tech Vocabulary
Computing Access
Motivation
Procrastination
Time Management
Help Seeking
Locus of Control
LEARNING STYLES
Visual
Verbal
Social
Solitary
Physical
Aural
Logical
TYPING
Rate
Accuracy
ON-SCREEN
READING
Rate
Recall

34. Adjusting Readiness Ranges
Adjusting the cut
points can make
the reporting a
more accurate
predictor of
success.

35. How Do Schools Use It?
•
•
•
•
•
•
•
Orientation Course
Enrollment Process
Information Webinar
Public Website
Class Participation
Facebook
68% of client schools administer the
assessment to all students, not just
eLearning students

36. Thermometer Analogy
• More important than taking your child’s
temperature is taking appropriate action
based on their temperature.
• More important than measuring student
readiness is taking appropriate action
based on the scores.

37. Progression of SmarterMeasure Data Utilization
Predictive
Correlation
Comparison
Descriptive
Student Service

38. Research
Ideas on the
Research
Page of the
Website

39. Results of Middlesex Research
Failure rates reduced by as much as 10%

40. The Findings
• Statistically Significant Relationships
Academic
Achievement
Engagement
Retention
Individual
Attributes
X
X
X
Technical
Knowledge
X
X
X
X
X
Learning
Styles
Life Factors
Technical
Competency
X
X
X

41. J. Sargeant Reynolds
Community College
• Required as admissions
assessment
• Integral part of their QEP
• Computed correlations
with grades and
Life
Factors
SmarterMeasure
sub-scales of over 4000
students.
• P
Attributes
Grades
Learning
Styles
Technical

42. Findings
• Statistically significant correlations:
- Dedicated place, support from employers
and family, access to study resources, and
academic skills (Life Factors)
- Tech vocabulary (Technical Knowledge)
- Procrastination (Individual Attributes)
Scores
Grades

43. Academic Success Rates
70
60
50
40
High Score
Low Score
30
20
10
0
Skills
Resources
Time
Less than 10% of students with low scores
experienced academic success.

44. Five Schools
What is the relationship between measures
of online student readiness and measures
of online student satisfaction?

45. Methodology
Incoming vs Outgoing
Data from 1,611 students who completed both the
SmarterMeasure Learning Readiness Indicator
and the Priority Survey for Online Learners were
analyzed.

46. Findings
• There were statistically significant
relationships between factors of readiness
and satisfaction.

47. SmarterMeasure.com

48. How important do you consider
non-cognitive skills?
How is your school measuring
and using non-cognitive
factors?

49. For More Info
SmarterServices.com
(877) 499-SMARTER
info@SmarterServices.com