Measuring What Matters
The Role of Non-Cognitive Factors
in Student Success
NUTN Webinar – February,
Dr. Mac Adkins, Presi...
Question 1?
• How do you determine who can be
enrolled at your school?
– Standardized test scores
– Prior grade point aver...
Top Admissions Factors
• The National Association for College Admission Counseling rated
these factors.
• CONSIDERABLY IMP...
Question 2
Why Do Students Drop Out?
A study funded by the Bill and Melinda Gates Foundation
ranked these reasons:
1. Conf...
To Find Out What Matters
Let’s Ask:
Employers
Colleges
Faculty
National Research Council
US Department of Education
Mother...
Skills that
Employer’s
Want
National Association of Colleges
and Employers
Survey of Employers

http://www.unl.edu/svcaa/d...
Outcomes Schools Want
Elements of Mission Statements From 35 Universities
1. Knowledge, learning, mastery of general princ...
Traits Online Faculty Want

WICHE Cooperative for Educational Technologies, 2013
2012 National Research Council
COGNITIVE
Problem solving
Critical thinking
Systems thinking
Study skills
Adaptability
Crea...
US Department of Education
―The test score accountability movement and
conventional educational approaches tend to
focus o...
Parents Teach It
Are You Beginning To See The Picture?
• Non-cognitive skills matter
– Determine student retention
– Determine employer sat...
―Years of schooling predicts labor market
outcomes — cognitive skills account for only
20%; therefore 80% of the ―years of...
Types of Data Used To Predict
Learner Success

APTITUDE

ATTITUDE

SITUATION
What Are Non-Cognitive Skills?
Can Non-Cognitive Skills
Be Taught?
You can’t
change a tiger’s
stripes, but you
can teach that
tiger to hunt in a
differen...
Recommended Uses of
Non-Cognitive Skills Measures
1. Optic – A lens through which students can view their
strengths and op...
Methods of Measurement
•
•
•
•
•

Instructor ratings – Time and task intensive for the faculty
Observer records – Expensiv...
Construct Comparison Matrix
ACT
Engage

Individual
Attributes

ETS Success
Navigator

Wonderlic
Admissions
Risk Profile

S...
SmarterMeasure
Learning Readiness Indicator
• A 124-item online skills test and attributes
inventory that measures a stude...
What Does The Assessment Measure?
INTERNAL

EXTERNAL

SKILLS

INDIVIDUAL
ATTRIBUTES

LIFE FACTORS

TECHNICAL

Availability...
Adjusting Readiness Ranges
Adjusting the cut
points can make
the reporting a
more accurate
predictor of
success.
How Do Schools Use It?
•
•
•
•
•
•
•

Orientation Course
Enrollment Process
Information Webinar
Public Website
Class Parti...
Thermometer Analogy
• More important than taking your child’s
temperature is taking appropriate action
based on their temp...
Progression of SmarterMeasure Data Utilization
Predictive
Correlation

Comparison
Descriptive
Student Service
Research
Ideas on the
Research
Page of the
Website
Results of Middlesex Research
Failure rates reduced by as much as 10%
The Findings
• Statistically Significant Relationships
Academic
Achievement

Engagement

Retention

Individual
Attributes
...
J. Sargeant Reynolds
Community College
• Required as admissions
assessment
• Integral part of their QEP
• Computed correla...
Findings
• Statistically significant correlations:
- Dedicated place, support from employers
and family, access to study r...
Academic Success Rates
70
60
50
40

High Score
Low Score

30
20
10
0
Skills

Resources

Time

Less than 10% of students wi...
Five Schools
What is the relationship between measures
of online student readiness and measures
of online student satisfac...
Methodology
Incoming vs Outgoing
Data from 1,611 students who completed both the
SmarterMeasure Learning Readiness Indicat...
Findings
• There were statistically significant
relationships between factors of readiness
and satisfaction.
SmarterMeasure.com
How important do you consider
non-cognitive skills?
How is your school measuring
and using non-cognitive
factors?
For More Info
SmarterServices.com
(877) 499-SMARTER
info@SmarterServices.com
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN
Upcoming SlideShare
Loading in …5
×

Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN

536 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
536
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Measuring What Matters; Noncognitive Skills - Webinar Hosted by NUTN

  1. 1. Measuring What Matters The Role of Non-Cognitive Factors in Student Success NUTN Webinar – February, Dr. Mac Adkins, President, SmarterServices Provided by
  2. 2. Question 1? • How do you determine who can be enrolled at your school? – Standardized test scores – Prior grade point averages – Admissions exams
  3. 3. Top Admissions Factors • The National Association for College Admission Counseling rated these factors. • CONSIDERABLY IMPORTANT – College prep course grades – Strength of high school curriculum – Standardized test scores – Overall GPA • MODERATELY IMPORTANT – Admissions essay – Letters of recommendation – Demonstrated interest – Class rank – Extracurricular commitment
  4. 4. Question 2 Why Do Students Drop Out? A study funded by the Bill and Melinda Gates Foundation ranked these reasons: 1. Conflict with work schedule 2. Affordability of tuition 3. Lack of support from family – financial and practical support 4. Lack of belief that a college degree is valuable 5. Lack of discipline – too much socializing, not enough studying http://www.publicagenda.org/pages/with-their-whole-lives-ahead-of-them
  5. 5. To Find Out What Matters Let’s Ask: Employers Colleges Faculty National Research Council US Department of Education Mothers
  6. 6. Skills that Employer’s Want National Association of Colleges and Employers Survey of Employers http://www.unl.edu/svcaa/documents/how_e mployers_see_candidates.pdf
  7. 7. Outcomes Schools Want Elements of Mission Statements From 35 Universities 1. Knowledge, learning, mastery of general principles 2. Continuous learning, intellectual interest, curiosity 3. Artistic cultural appreciation 4. Appreciation for diversity 5. Leadership 6. Interpersonal skills 7. Social responsibility, citizenship and involvement 8. Physical and psychosocial health 9. Career preparation 10.Adaptability and life skills 11.Perseverance 12.Ethics and integrity Michigan State University, 2004
  8. 8. Traits Online Faculty Want WICHE Cooperative for Educational Technologies, 2013
  9. 9. 2012 National Research Council COGNITIVE Problem solving Critical thinking Systems thinking Study skills Adaptability Creativity Meta-cognitive skills INTERPERSONAL Communication Social Intelligence Teamwork Leadership Cultural sensitivity Tolerance for diversity INTRAPERSONAL Anxiety Self-efficacy Self-concept Attributions Work ethic Persistence Organization Time management Integrity Life-long learning
  10. 10. US Department of Education ―The test score accountability movement and conventional educational approaches tend to focus on intellectual aspects of success, such as content knowledge. However, this is not sufficient. If students are to achieve their full potential, they must have opportunities to engage and develop a much richer set of skills. There is a growing movement to explore the potential of the ―noncognitive‖ factors — attributes, dispositions, social skills, attitudes, and intrapersonal resources, independent of intellectual ability—that high-achieving individuals draw upon to accomplish success.‖
  11. 11. Parents Teach It
  12. 12. Are You Beginning To See The Picture? • Non-cognitive skills matter – Determine student retention – Determine employer satisfaction – Determine online course success – Federal agencies recognize their importance – They are the mission of many schools – Parents value them
  13. 13. ―Years of schooling predicts labor market outcomes — cognitive skills account for only 20%; therefore 80% of the ―years of schooling‖ benefit is due to noncognitive skills‖ (Bowles, Gintis, & Osborne, 2001) http://www.umass.edu/preferen/gintis/jelpap.pdf
  14. 14. Types of Data Used To Predict Learner Success APTITUDE ATTITUDE SITUATION
  15. 15. What Are Non-Cognitive Skills?
  16. 16. Can Non-Cognitive Skills Be Taught? You can’t change a tiger’s stripes, but you can teach that tiger to hunt in a different environment.
  17. 17. Recommended Uses of Non-Cognitive Skills Measures 1. Optic – A lens through which students can view their strengths and opportunities for improvement 2. Student Service – A tool to guide students toward available resources for support 3. Placement – Developmental / remedial course placement 4. Talking Points – A collection of statements which academic advisors can use to advise their students 5. Early Alert – A list of students who are likely to be benefitted by the instructor reaching out to them early in the course. 6. Predictive Analytic - A set of data which can be analyzed at the individual and aggregate level to project student performance
  18. 18. Methods of Measurement • • • • • Instructor ratings – Time and task intensive for the faculty Observer records – Expensive and time consuming Letters of recommendation – Rarely objective Interviews – Time consuming to conduct and code Socioeconomic data – Beneficial mostly at the aggregate level due to exceptions and bias • Self assessment – Yes, there are limitations, but it is the preferred method.
  19. 19. Construct Comparison Matrix ACT Engage Individual Attributes ETS Success Navigator Wonderlic Admissions Risk Profile SmarterMeasure X X X X Life Factors X Learning Styles X Technical Skills X X Reading Skills X Keyboarding Skills X Custom Questions X
  20. 20. SmarterMeasure Learning Readiness Indicator • A 124-item online skills test and attributes inventory that measures a student’s level of readiness for studying online • Used by over 500 Colleges and Universities • Since 2002 taken by over 2,600,000 students
  21. 21. What Does The Assessment Measure? INTERNAL EXTERNAL SKILLS INDIVIDUAL ATTRIBUTES LIFE FACTORS TECHNICAL Availability of Time Dedicated Place Reason Support from Family Technology Usage Life Application Tech Vocabulary Computing Access Motivation Procrastination Time Management Help Seeking Locus of Control LEARNING STYLES Visual Verbal Social Solitary Physical Aural Logical TYPING Rate Accuracy ON-SCREEN READING Rate Recall
  22. 22. Adjusting Readiness Ranges Adjusting the cut points can make the reporting a more accurate predictor of success.
  23. 23. How Do Schools Use It? • • • • • • • Orientation Course Enrollment Process Information Webinar Public Website Class Participation Facebook 68% of client schools administer the assessment to all students, not just eLearning students
  24. 24. Thermometer Analogy • More important than taking your child’s temperature is taking appropriate action based on their temperature. • More important than measuring student readiness is taking appropriate action based on the scores.
  25. 25. Progression of SmarterMeasure Data Utilization Predictive Correlation Comparison Descriptive Student Service
  26. 26. Research Ideas on the Research Page of the Website
  27. 27. Results of Middlesex Research Failure rates reduced by as much as 10%
  28. 28. The Findings • Statistically Significant Relationships Academic Achievement Engagement Retention Individual Attributes X X X Technical Knowledge X X X X X Learning Styles Life Factors Technical Competency X X X
  29. 29. J. Sargeant Reynolds Community College • Required as admissions assessment • Integral part of their QEP • Computed correlations with grades and Life Factors SmarterMeasure sub-scales of over 4000 students. • P Attributes Grades Learning Styles Technical
  30. 30. Findings • Statistically significant correlations: - Dedicated place, support from employers and family, access to study resources, and academic skills (Life Factors) - Tech vocabulary (Technical Knowledge) - Procrastination (Individual Attributes) Scores Grades
  31. 31. Academic Success Rates 70 60 50 40 High Score Low Score 30 20 10 0 Skills Resources Time Less than 10% of students with low scores experienced academic success.
  32. 32. Five Schools What is the relationship between measures of online student readiness and measures of online student satisfaction?
  33. 33. Methodology Incoming vs Outgoing Data from 1,611 students who completed both the SmarterMeasure Learning Readiness Indicator and the Priority Survey for Online Learners were analyzed.
  34. 34. Findings • There were statistically significant relationships between factors of readiness and satisfaction.
  35. 35. SmarterMeasure.com
  36. 36. How important do you consider non-cognitive skills? How is your school measuring and using non-cognitive factors?
  37. 37. For More Info SmarterServices.com (877) 499-SMARTER info@SmarterServices.com

×