The document discusses foundational concepts in assessment including things and stuff, variables, prediction and control, and relationships. It uses examples to explain these concepts. Things refer to entities being assessed, while stuff describes their properties. Properties are represented by variables that can be measured precisely or not. The broad goals of assessment are to understand populations by predicting variables and relationships between entities and properties. Understanding relationships is important for accurate prediction and control, which allows for better understanding.
Assessment Reconsidered: Why direct measures are important and how to impleme...Stan Dura
Presentation on the need and benefits of using direct measures in student affairs to capture rich evidence of student learning and competencies in co-curricular experiences.
RedTacton is a Human Area Networking technology which uses the surface of the human body as a safe, high speed network transmission path. It is completely distinct from wireless and infrared technologies as it uses the minute electric field emitted on the surface of the human body.
Assessment Reconsidered: Why direct measures are important and how to impleme...Stan Dura
Presentation on the need and benefits of using direct measures in student affairs to capture rich evidence of student learning and competencies in co-curricular experiences.
RedTacton is a Human Area Networking technology which uses the surface of the human body as a safe, high speed network transmission path. It is completely distinct from wireless and infrared technologies as it uses the minute electric field emitted on the surface of the human body.
Business Research Method - Unit III, AKTU, Lucknow SyllabusKartikeya Singh
Business Research Method - Unit III, AKTU, Lucknow Syllabus,
Research Methodology - Topics Covered - Scaling & Measurement techniques: Concept of Measurement: Need of Measurement; Problems in measurement in management research – Validity and Reliability. Levels of measurement – Nominal, Ordinal, Interval, Ratio.
Attitude Scaling Techniques: Concept of Scale – Rating Scales viz. Likert Scales, Semantic Differential Scales, Constant Sum Scales, Graphic Rating Scales – Ranking Scales – Paired comparison & Forced Ranking – Concept and Application.
Why limit ourselves to traditional quantitative metrics like visitor count, page weight, conversion, and revenue when there is so much valuable qualitative data available? We can turn qualitative data into quantitative data and use the same rigorous analysis techniques to help lead us to better designs, products, services, and experiences.
Dr. Lani discusses all aspects of the dissertation methodology, including: selecting a survey instrument, population, reliability, validity, data analysis plan, and IRB/URR considerations.
A very interesting and enjoyable seeion with Pia and Piyul, knowing the mind-set of the people in a rural setup and how different it is from the urban society.
For more details read our blog :
DiSC Profile Insight offers its buyers a step by step overview of what their profiles mean; how to read them, how to interpret the graphs and translate what values and assets can be taken out of the report to better your work environment, and communication.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
More Related Content
Similar to Student Affairs Assessment Committee Training
Business Research Method - Unit III, AKTU, Lucknow SyllabusKartikeya Singh
Business Research Method - Unit III, AKTU, Lucknow Syllabus,
Research Methodology - Topics Covered - Scaling & Measurement techniques: Concept of Measurement: Need of Measurement; Problems in measurement in management research – Validity and Reliability. Levels of measurement – Nominal, Ordinal, Interval, Ratio.
Attitude Scaling Techniques: Concept of Scale – Rating Scales viz. Likert Scales, Semantic Differential Scales, Constant Sum Scales, Graphic Rating Scales – Ranking Scales – Paired comparison & Forced Ranking – Concept and Application.
Why limit ourselves to traditional quantitative metrics like visitor count, page weight, conversion, and revenue when there is so much valuable qualitative data available? We can turn qualitative data into quantitative data and use the same rigorous analysis techniques to help lead us to better designs, products, services, and experiences.
Dr. Lani discusses all aspects of the dissertation methodology, including: selecting a survey instrument, population, reliability, validity, data analysis plan, and IRB/URR considerations.
A very interesting and enjoyable seeion with Pia and Piyul, knowing the mind-set of the people in a rural setup and how different it is from the urban society.
For more details read our blog :
DiSC Profile Insight offers its buyers a step by step overview of what their profiles mean; how to read them, how to interpret the graphs and translate what values and assets can be taken out of the report to better your work environment, and communication.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
The Indian economy is classified into different sectors to simplify the analysis and understanding of economic activities. For Class 10, it's essential to grasp the sectors of the Indian economy, understand their characteristics, and recognize their importance. This guide will provide detailed notes on the Sectors of the Indian Economy Class 10, using specific long-tail keywords to enhance comprehension.
For more information, visit-www.vavaclasses.com
Model Attribute Check Company Auto PropertyCeline George
In Odoo, the multi-company feature allows you to manage multiple companies within a single Odoo database instance. Each company can have its own configurations while still sharing common resources such as products, customers, and suppliers.
How to Split Bills in the Odoo 17 POS ModuleCeline George
Bills have a main role in point of sale procedure. It will help to track sales, handling payments and giving receipts to customers. Bill splitting also has an important role in POS. For example, If some friends come together for dinner and if they want to divide the bill then it is possible by POS bill splitting. This slide will show how to split bills in odoo 17 POS.
How to Make a Field invisible in Odoo 17Celine George
It is possible to hide or invisible some fields in odoo. Commonly using “invisible” attribute in the field definition to invisible the fields. This slide will show how to make a field invisible in odoo 17.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasn’t one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
2. Training Agenda
• Research, Assessment, & Program Evaluation
• Foundational Concepts
– Measurement is imprecise
– “Things and the stuff about them”
– Variables, variance, and modeling
• Q & A
10. Surveys are especially vulnerable
What
“construct” are
you measuring?
Survey and
question design
and content
Responses
11. Surveys are especially vulnerable
What
“construct” are
you measuring?
Survey and
question design
and content
Responses
Is it really measuring it?
Does it measure it the same
each time?
12. Surveys are especially vulnerable
What
“construct” are
you measuring?
Survey and
question design
and content
Responses
Is it really measuring it?
Does it measure it the same
each time?
Any errors in the questions or
scales?
Any bias in the questions
asked or not asked?
13. Surveys are especially vulnerable
What
“construct” are
you measuring?
Survey and
question design
and content
Responses
Is it really measuring it?
Does it measure it the same
each time?
Any errors in the questions or
scales?
Any bias in the questions
asked or not asked?
Response set–
-Are they representative?
-How complete are they?
-Is there non-response bias?
14. Surveys are especially vulnerable
What
“construct” are
you measuring?
Survey and
question design
and content
Responses
Is it really measuring it?
Does it measure it the same
each time?
Any errors in the questions or
scales?
Any bias in the questions
asked or not asked?
Response set–
-Are they representative?
-How complete are they?
-Is there non-response bias?
Over 80
cognitive and
memory
biases
15. Respondents prone to error
• Response is a cognitive act. 4 Stages:
• Comprehension
• Retrieval
• Judgment
• Response
All kinds of errors can happen in these stages
16. Respondents prone to error
Cognitive Stages Definition Vulnerabilities
Stage 1 Comprehension
Understanding the question
as author intended
Unknown or misused words,
ambiguity, complexity, length
Stage 2 Retrieval
Search of memory for
relevant information
Memory bias, recall error,
fatigue
Stage 3 Judgment
Considers information
retrieved, makes
“guestimations” and decides
Social or political bias, “fuzzy
logic,” error in “guesstimation”,
personal sensitivity
Stage 4 Response Provides the information
Human error, incomplete
response, wrong format
17. Respondents prone to error
• Change bias – Remembers as more difficult
• Context effect – out of context more difficult
• Consistency bias – thinking past attitudes same
as current
• Anchoring – relying too heavily on one piece of
info in order to make an estimation
• Availability heuristic – overestimation due to
recency or emotional strength of memory
18. Respondents prone to error
• Confirmation bias – tendency to search for and
remember info confirming one’s preconceptions
• Framing effect – drawing different conclusions
or interpretations base on how info is presented
• Optimism bias – tendency to overestimate
likelihood of positive outcomes
• Social response bias – tendency to underreport
socially undesirable behaviors and vice/versa.
Lots more!!!!
19. Proof is in the Polls
• Ever seen this?
• Down in the polls by 6 points
• Down in the polls by 3 points
• Ahead in the polls by 1 point
• Landslide victory with 60% of the votes!
Keep these limitations in mind when drawing
inferences from survey data.
20. Research, Assessment, & Program
Evaluation
Research
• Systematic collection of data
• Generalize to larger populations
• Based on Research question
• Rigorous methodology
• Used to test hypotheses
• Contributes to the development
of theory and models
Assessment
• Systematic collection of data
• Generalize to larger population
• Based on Assessment question
• Borrows rigor when practical
• Used to evaluate effectiveness
• Contributes to judgments of the
quality of programs or activities
21. Program Evaluation:
• Act of using data (assessment, operational,
etc.) related to outcomes, inputs, and
processes of a program
• To assign judgment on its effectiveness
Research, Assessment, & Program
Evaluation
22. Assessment (and statistics) is about describing:
• Things
• Stuff pertaining to those things
• The relationship between things & stuff
Foundational Concepts
“Things and Stuff”
23. What is a “thing”?
• An entity of some kind
• An idea
• A quality or characteristic
Foundational Concepts
“Things and Stuff”
24. What is “stuff”?
• Material out of which something is made
• Essential substance or elements
• Essence
Foundational Concepts
“Things and Stuff”
25. Things are the objects we assess,
and
Stuff characterizes those things
Foundational Concepts
“Things and Stuff”
26. Let’s use more concrete terms:
• Entity – something that exists separate from its
parts
• Property – a characteristic, trait or attribute and
other stuff that describes something
Foundational Concepts
“Things and Stuff”
28. Different Properties:
– Height
– Age
– Perceptions
– Weight
– Density
– GPA
– Level of engagement
– Magnitude
– Levels and expressions of anger, fear, anxiety
Foundational Concepts
“Things and Stuff”
29. Notice that some properties could be examined
as entities themselves
– Relaxation
– Motivation
– Intelligence
– Satisfaction
– Health
Foundational Concepts
“Things and Stuff”
30. What entities are we concerned with?
Foundational Concepts
“Things and Stuff”
31. What properties of those entities concern us?
Foundational Concepts
“Things and Stuff”
32. What properties of those entities concern us?
Foundational Concepts
“Things and Stuff”
33. Entities and properties
– Humans
• Height, satisfaction, health, perceptions, emotions, beliefs,
etc.
– Students (whether human or not )
• Age, GPA, knowledge, skills, engagement, class standing, etc.
– Physical Objects
• Weight, chemical composition, density, etc.
– Forces
• Magnitude, direction, etc.
Foundational Concepts
“Things and Stuff”
34. Most critical thing about properties -
they vary
– Height varies
– Age varies
– Perceptions vary
– Weight varies
– Density varies
– GPA varies
– Level of engagement varies
– Levels and expressions of anger, fear, anxiety vary
Foundational Concepts
“Things and Stuff”
35. These are the first two foundational concepts:
• Entity
• Property
The others are:
• Variable
• Prediction and Control
• Relationship
• Statistical techniques
Foundational Concepts
“Things and Stuff”
36. Properties are represented by
variables -
– Height is represented by inches, meters, or relational terms
(bigger)
– Age is represented by months, days, years or relational terms
(older)
– Weight is represented by kilograms, pounds or relational terms
– Density is represented by pound per square inch, etc.
– Grades can be represented by symbols (A+), numbers (92), etc.
– Engagement can be represented by number of activities
attended, etc.
Foundational Concepts
“Things and Stuff”
37. Some variables are straightforward-
–Height is often measured in Inches
–Weight is often measured in Kilograms
–Age is often measured in years
Foundational Concepts
“Things and Stuff”
38. Some variables are not-
–Satisfaction is often measured in… ???
–Anger is often measured in… ???
–Engagement is often measured in… ???
Foundational Concepts
“Things and Stuff”
39. Thus variables are just
representations
of the properties
Foundational Concepts
“Things and Stuff”
40. That describe an entity
to some degree
Foundational Concepts
“Things and Stuff”
41. That describe an entity to
some degree
That may be precise or not
Foundational Concepts
“Things and Stuff”
42. Broad goal of assessment/research:
To understand the population
Two underlying fundamental goals:
– Prediction and control of variables
– Understanding of entities & their properties
Which is more important?
Foundational Concepts
Prediction and Control
43. Let’s pretend we are looking at student
performance of a complex task:
Let’s look at these statements:
– Haste makes waste
– Practice makes perfect
How would we assess these?
Foundational Concepts
Prediction and Control
44. What are our variables?
– What variables might represent haste?
– What variables might represent waste?
– What variables might represent practice?
– What variables might represent perfection?
What is our model?
Foundational Concepts
Prediction and Control
45. What is our model?
–More haste = more waste?
• Some of the waste is the result of haste?
• Haste predicts waste?
– Simply looking at means, which is what we often do:
• Those that were hastier answered fewer questions
correctly than those who took more time.
Foundational Concepts
Prediction and Control
47. Waste
Haste
• The red area represents the degree to which Waste is due to Haste.
• The green area is the degree to which Waste is influenced by other factors
Foundational Concepts
Prediction and Control
48. What is our model?
–More practice = better performance?
• Some of the performance is the result of practice?
• Practice predicts performance?
– Simply looking at means, which is what we often do:
• Those who practiced more scored higher than those who
practiced less.
Foundational Concepts
Prediction and Control
49. Performan
ce
• What is the red area?
• What is the green area?
Practice
Foundational Concepts
Prediction and Control
50. Is it really that simple?
–What if hasty practice results in worse
performance?
–How do we know the true impact of practice,
if we don’t remove the impact of haste?
Foundational Concepts
Prediction and Control
51. So in order to accurately understand
entities and properties, we must be
able to predict and control.
–Prediction and control are more important –
accurate understanding depends on it
–We must understand relationships to do so
Foundational Concepts
Prediction and Control
52. Back to those statements
– Haste makes waste
– Practice makes perfect
Both are engrained in our culture and both are
about relationships
Establishing relationships between objects is how
we naturally understand the world
Foundational Concepts
Prediction and Control
53. Let’s operationalize this?
–Haste = time on task (in minutes)
–Practice = # of repetitions
–Performance = Evaluation score
• (waste = lost points or distance from a perfect score)
Foundational Concepts
Prediction and Control
54. Our Model
• Performance = Practice + Haste
Or more specifically…
• Evaluation Score = # Repetitions + Minutes on Task
Foundational Concepts
Prediction and Control
55. Each variable contributes to the
variance in evaluation scores
scores
#
repetitions
minutes
on task
Foundational Concepts
Prediction and Control
56. scores extent to which
observed scores vary
across all individuals
We need to control for the impact of Haste on Performance
in order to understand the impact of Practice.
Foundational Concepts
Prediction and Control
58. extent to
which minutes
on task varies
across all
individuals
Foundational Concepts
Prediction and Control
scores
#
repetitions
minutes
on task
59. Foundational Concepts
Prediction and Control
scores
#
repetitions
minutes
on task
variance in scores that we
can’t explain (individual
differences, fatigue, human
error, etc.)
65. Foundational Concepts
Prediction and Control
scores
#
repetitions
minutes
on task
variance in Scores
attributed to the
combination of
minutes on task AND #
repetitions
67. Foundational Concepts
Prediction and Control
scores
#
repetitions
minutes
on task
Put another way:
variance in scores
attributed to #
repetitions AFTER
variance
attributed to
minutes on task is
controlled
70. If we did not Control
for Haste, we would
think all of this was
the result of Practice
Performance
Practice
Foundational Concepts
Prediction and Control
71. When really, this was
the impact.
Performance
Practice
Foundational Concepts
Prediction and Control
72. So…
• Recognizing the relationships between
variables, and
• controlling for them
• is critical to understanding the impact
of any one of them.
Foundational Concepts
Prediction and Control
73. Why is this important in assessment?
• Consider you are studying the impact of stress on student
persistence?
• You ask students to describe their stress level at one point
in time, and you see that:
– 80% of those who reported high stress dropped out, while only
– 25% of those who reported low stress did.
• You devote 70% of a staff person’s time to doing stress
relief programming, totaling appx. $60,000
Foundational Concepts
Prediction and Control
74. What variables are there in stress and
persistence?
Stress
• Family support and relationships
• Peer support and relationships
• Coping skills
• Academic work load
• Financial Support
• Employment work load
• Work environment
Persistence
• Academic advising
• Social connectedness
• Engagement
• Faculty / Staff connections
• Institutional Bureaucracy
• Meaningful learning experiences
• Support Services
Foundational Concepts
Prediction and Control
75. Stress
• Family support and relationships
• Peer support and relationships
• Coping skills
• Academic work load
• Financial Support
• Employment work load
• Work environment
Persistence
• Academic advising
• Social connectedness
• Engagement
• Faculty / Staff connections
• Institutional Bureaucracy
• Meaningful learning experiences
• Support Services
• What should the staff member focus their efforts on?
• Is it possible that general Stress Relief may not be a
very influential variable?
• Would the $60k be well spent?
Foundational Concepts
Prediction and Control
76. When we conduct quantitative or qualitative
analyses, we are constructing models
• That model can be very, very simple
• Mean – this is how the average person did…
• Mode – The most frequent score was X
• Standard Deviation – roughly 68% scored between these two
points
• Range – All students scored between X and Y.
• Descriptive stats = the weakest and least informative models
Foundational Concepts
Prediction and Control
77. When we conduct quantitative or qualitative
analyses, we are constructing models
• Other models can be complex
– For every 1 repetition of Practice, when Haste is controlled, Evaluation
Scores increased by 2.4 points.
– For every hour engaged in co-curricular activities, when controlling for HS
GPA, Class standing, and Major, retention likelihood increases by .45%
– More complex stats = more informative and more
accurate models
Foundational Concepts
Prediction and Control
78. So when we assess, we need to
Think of our Model
• What are the entities and properties we’re examining?
• What are the variables involved? Operationalize them.
• What is their relationship; how do they interact?
• How simple or complex/accurate a model do I need?
• How can we predict and control for these variables?
Foundational Concepts
Prediction and Control