This lecture recaps the previous lecture on exploratory factor analysis, and introduces psychometrics and (fuzzy) concepts and their measurement, including (operationalisation), reliability (particularly internal consistency of multi-item measures), validity and the creation of composite scores. See also https://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Lectures/Psychometric_instrument_development
Definition
A procedure used to collect both qualitative and quantitative data.
This is done due to the fact that it is believed that both types of studies will provided a clearer understanding of what is being studied.
“It consists of merging ,integrating ,linking ,or embedding the two “strands””(Ceswell,2012).
Definition
A procedure used to collect both qualitative and quantitative data.
This is done due to the fact that it is believed that both types of studies will provided a clearer understanding of what is being studied.
“It consists of merging ,integrating ,linking ,or embedding the two “strands””(Ceswell,2012).
Qualitative methods in Psychology ResearchDr. Chinchu C
An introduction to Qualitative Methods in Psychology. Intended mostly for UG/PG students. Conveys the essentials of Ontology and Epistemology and moves on to the popular methods in Qualitative Psychological Research
A presentation about the added value of combining qualitative and quantitative methods. It begins with a brief discussion of qualitative research and how it is distinct from yet shares basic principles with quantitative research, followed by a discussion of four important ways mixed methods -- integrating qualitative and quantitative -- adds value to our research efforts, and then a discussion of mixed methods research -- what it is, typologies, alternatives to typologies, and the use of diagrams.
The present ppt will help the student to get the idea of response set or bias. The ppt will help the learner to understand various types of biases and techniques for controlling them.
Dr. Lani discusses all aspects of the dissertation methodology, including: selecting a survey instrument, population, reliability, validity, data analysis plan, and IRB/URR considerations.
Qualitative methods in Psychology ResearchDr. Chinchu C
An introduction to Qualitative Methods in Psychology. Intended mostly for UG/PG students. Conveys the essentials of Ontology and Epistemology and moves on to the popular methods in Qualitative Psychological Research
A presentation about the added value of combining qualitative and quantitative methods. It begins with a brief discussion of qualitative research and how it is distinct from yet shares basic principles with quantitative research, followed by a discussion of four important ways mixed methods -- integrating qualitative and quantitative -- adds value to our research efforts, and then a discussion of mixed methods research -- what it is, typologies, alternatives to typologies, and the use of diagrams.
The present ppt will help the student to get the idea of response set or bias. The ppt will help the learner to understand various types of biases and techniques for controlling them.
Dr. Lani discusses all aspects of the dissertation methodology, including: selecting a survey instrument, population, reliability, validity, data analysis plan, and IRB/URR considerations.
Paul Gerrard - Advancing Testing Using Axioms - EuroSTAR 2010TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on Advancing Testing Using Axioms by Paul Gerrard. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
A critical review of the nowadays practices in work-related stress risk assessment and reduction. With a case history.
It was presented in the event "Work Related Stress: How to prevent,
assess and tackle the risk at European scale" organised by the CIIP (Consulta Interassociativa Italiana per la prevenzione) on Oct. 23, 2015 in Milan. The event was closing the CIIP initiatives in the European 2015 Campaign on work-related stress.
All the concepts related to research design are covered in this PPT Presentation.Research Design being an integral and crucial part of Research majorly deals with Parametric and non-parametric test, Type 1 and type 2 error, level of significance etc.It helps in ascertaining which research technique is used in which situation.
Development and evaluation of the PCYC Catalyst outdoor adventure interventio...James Neill
Neill, J. T., Sullivan, A., Bowen, D. J. (2016). Development and evaluation of the PCYC Catalyst outdoor adventure intervention program for youth-at-risk. Paper presented at the 19th National Outdoor Education Conference, University of Sunshine Coast, Sippy Downs, Queensland, March 29 - April 1.
A 15 minute session presented during Open Access Week 2014 at University of Canberra on Monday 20 October 12:30-1:00 Project Room 3 (1C77 - Student Commons)
Explains some advanced uses of multiple linear regression, including partial correlations, analysis of residuals, interactions, and analysis of change. See also previous lecture http://www.slideshare.net/jtneill/multiple-linear-regression
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasn’t one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
4. 4
What is factor analysis?
• Factor analysis is:
–a family of multivariate correlational
methods used to identify clusters of
covariance (called factors)
• Two main purposes:
–Theoretical (PAF)
–Data reduction (PC)
• Two main types (extraction methods):
–Exploratory factor analysis (EFA)
–Confirmatory factor analysis (CFA)
5. 5
EFA steps
1. Test assumptions
– Sample size
• 5+ cases x no. of variables (min.)
• 20+ cases x no. of variables (ideal)
• Another guideline: Or N > 200
– Outliers & linearity
– Factorability
• Correlation matrix: Some > .3?
• Anti-image correlation matrix diags > .5
• Measures of Sampling Adequacy: KMO > ~ .5 to 6;
Bartlett's sig?
6. 6
EFA steps
2. Select type of analysis
– Extraction
• Principal Components (PC)
• Principal Axis Factoring (PAF)
– Rotation
• Orthogonal (Varimax)
• Oblique (Oblimin)
7. 7
Summary of EFA steps
3. Determine no. of factors
– Theory?
– Kaiser's criterion?
– Eigen Values and Scree plot?
– % variance explained?
– Interpretability of weakest factor?
8. 8
Summary of EFA steps
4. Select items
– Use factor loadings to help identify which items
belong in which factor
– Drop items one at a time if they don't belong to any
factor e.g., consider any items for which
• primary (highest) loading is low? (< .5 ?)
• cross- (other) loading(s) are high? (> .3 ?)
• item wording doesn't match the meaning of the factor
9. 9
Summary of EFA steps
5. Name and describe factors
6. Examine correlations amongst
factors
7. Analyse internal reliability
8. Compute composite scores
9. Check factor structure across
sub-groups
Covered in
this lecture
11. 11
● 271 UC students responded to 24
university student motivation
statements in 2008 using an 8-point
Likert scale (False to True)
● For example:
“I study at university … ”
– to enhance my job prospects.
– because other people have told me I should.
● EFA PC Oblimin revealed 5 factors
Example EFA:
University student motivation
15. 15
Example EFA:
University student motivation factor correlations
Motivation Self
Develo
pment
Social
Enjoy
ment
Altruis
m
Social
Pressure
Career &
Qualifications
.26 .25 .24 .06
Self Development .33 .55 -.18
Social Enjoyment .26 .33
Altruism .11
20. 20
Concepts & their measurement:
Bryman & Cramer (1997)
Concepts
• express common elements in the world
(to which we give a name)
• form a linchpin in the process of social
research
21. 21
Concepts & their measurement:
Bryman & Cramer (1997)
Hypotheses
• specify expected relations between
concepts
22. 22
Concepts & their measurement:
Bryman & Cramer (1997)
Operationalisation
• A concept needs to be operationally
defined in order for systematic research
to be conducted in relation to it.
• “An operational definition specifies the
procedures (operations) that will permit
differences between individuals in respect of
the concept(s) concerned to be precisely
specified ..."
23. 23
Concepts & their measurement:
Bryman & Cramer (1997)
“... What we are in reality talking
about here is measurement, that is,
the assignment of numbers to the
units of analysis - be they people,
organizations, or nations - to which a
concept refers."
24. Operationalisation
● The act of making a
fuzzy concept
measurable.
● Social science often
uses multi-item
measures to assess
related but distinct
aspects of a fuzzy
concept.
25. 25
Operationalisation steps
1. Brainstorm indicators of a concept
2. Define the concept
3. Draft measurement items
4. Pre-test and pilot test
5. Examine psychometric properties
- how precise are the measures?
6. Redraft/refine and re-test
30. 30
Measurement error
Measurement error is statistical
deviation from the true value caused
by the measurement procedure.
• Observed score =
true score +/- measurement error
• Measurement error =
systematic error +/- random error
● Systematic error =
• sampling error +/- non-sampling error
31. 31
Systemic and random
error
Measurement errorSystematic error – e.g., bathroom scales
aren't calibrated properly, and every measurement is
0.5kg too high. The error occurs for each
measurement.
Random error – e.g., measure your weight 3
times using the same scales but get three slightly
different readings. The error is different for each
measurement.
Image source: http://www.freestockphotos.biz/stockphoto/17079
32. 32
Sources of systematic error
Test reliability &
validity
(e.g., unreliable
or invalid tests)
Sampling
(non-
representative
sample)
Researcher bias
(e.g., researcher
favours a hypothesis)
Paradigm
(e.g., focus on positivism)
Respondent bias
(e.g., social desirability)
Non-sampling
33. 33
Measurement precision & noise
• The lower the precision, the more
participants are needed to make up for
the "noise" in the measurements.
• Even with a larger sample, noisy data
can be hard to interpret.
• Especially when testing and assessing
individual clients, special care is needed
when interpreting results of noisy tests.
http://www.sportsci.org/resource/stats/precision.html
34. 34
Minimising measurement error
• Standardise administration
conditions with clear instructions
and questions
• Minimise potential demand
characteristics (e.g., train interviewers)
• Use multiple indicators for fuzzy
constructs
35. 35
Minimising measurement error
• Obtain a representative sample:
–Use probability-sampling, if possible
–For non-probability sampling, use
strategies to minimise selection bias
• Maximise response rate:
–Pre-survey contact
–Minimise length / time / hassle
–Rewards / incentives
–Coloured paper
–Call backs / reminders
36. 36
• Ensure administrative accuracy:
–Set up efficient coding, with well-
labelled variables
–Check data (double-check at least a
portion of the data)
Minimising measurement error
38. 38
Psychometrics: Goal
To validly measure differences
between individuals and groups in
psychosocial qualities such as
attitudes and personality.
39. 39
Psychometric tasks
• Develop approaches and
procedures (theory and practice)
for measuring psychological
phenomena
• Design and test psychological
measurement instrumentation
(e.g., examine and improve reliability and
validity of psychological tests)
40. 40
Psychometrics: As test-taking
grows, test-makers grow rarer
"Psychometrics, one of the most
obscure, esoteric and cerebral
professions in America, is now also
one of the hottest.”
- As test-taking grows, test-makers grow rarer, David M. Herszenhor, May
5, 2006, New York Times
Psychometricians are in demand due to
increased testing of educational and
psychological capacity and performance.
43. 43
Reliability and validity
(Howitt & Cramer, 2005)
Reliability and validity (“classical test
theory”) are ways of evaluating the
accuracy of psychological tests and
measures.
• Reliability is about consistency of
– items within the measure
– the measure over time
• Validity is about whether the measure
actually measures what it is intended to
measure.
44. Reliability vs. validity
In classical test theory, reliability is generally thought to be
necessary for validity, but it does not guarantee validity.
In practice, a test of a relatively changeable psychological
construct such as suicide ideation, may be valid (i.e.,
accurate), but not particularly reliable over time (because
suicide ideation is likely to fluctuate).
45. Reliability vs. validity
Reliability
• A car which starts every time is reliable.
• A car which does not start every time is unreliable.
Validity
• A car which always reaches the desired destination is valid.
• A car which misses the desired destination is not valid.
Image source: https://commons.wikimedia.org/wiki/File:Aiga_carrental_cropped.svg
46. 46
Reliability and validity
(Howitt & Cramer, 2005)
• Reliability and validity are not
inherent characteristics of
measures. They are affected by the
context and purpose of the
measurement → a measure that is
valid for one purpose may not be
valid for another purpose.
48. 48
Types of reliability
• Internal consistency: Correlation
among multiple items in a factor
– Cronbach's Alpha (α)
• Test-retest reliability: Correlation
between test at one time and another
– Product-moment correlation (r)
• Inter-rater reliability: Correlation
between one observer and another:
– Kappa
49. 49
Reliability rule of thumb
< .6 = Unreliable
.6 = OK
.7 = Good
.8 = Very good, strong
.9 = Excellent
> .95 = may be overly reliable or
redundant – this is subjective and depends
on the nature what is being measured
50. Reliability rule of thumb
Table 7 Fabrigar et al (1999).
Table 7 Fabrigar et al. (1999)
Rule of thumb - reliability coefficients should be over .70, up to approx. .95
51. 51
Internal consistency
(or internal reliability)
Internal consistency refers to:
• How well multiple items combine as a
measure of a single concept
• The extent to which responses to multiple
items are consistent with one another
Internal consistency can measured by:
• Split-half reliability
• Odd-even reliability
• Cronbach's Alpha (α)
53. 53
Types of internal consistency:
Split-half reliability
• Sum the scores for the first half
(e.g., 1, 2, 3) of the items.
• Sum the scores for the second
half (e.g., 4, 5, 6) of the items.
• Compute a correlation between
the sums of the two halves.
54. 54
Types of internal consistency:
Odd-even reliability
• Sum the scores for items
(e.g.,1, 3, 5)
• Sum the scores for items
(e.g., 2, 4, 6)
• Compute a correlation between
the sums of the two halves.
55. 55
Types of internal reliability:
Alpha (Cronbach's α)
• Averages all possible split-half
reliability coefficients.
• Akin to a single score which
represents the degree of
intercorrelation amongst the items.
• Most commonly used indicator of
internal reliability
56. 56
• More items → greater reliability
(The more items, the more ‘rounded’ the
measure)
• Minimum items to create a factor is 2.
• There is no maximum, however the law
of diminishing returns is that each
additional item will add less and less to
the reliability.
• Typically ~ 4 to 12 items per factor are
used.
• Final decision is subjective and
depends on research context
How many items per factor?
57. 57
Internal reliability example:
Student-rated
quality of maths teaching
• 10-item scale measuring students’
assessment of the educational
quality of their maths classes
• 4-point Likert scale ranging from:
strongly disagree to strongly agree
58. 58
Quality of mathematics teaching
1. My maths teacher is friendly and cares
about me.
2. The work we do in our maths class is
well organised.
3. My maths teacher expects high
standards of work from everyone.
4. My maths teacher helps me to learn.
5. I enjoy the work I do in maths classes.
+ 5 more
61. SPSS: Cronbach’s α
If Cronbach's α if item
deleted is higher than the
α, consider removing
item.
62. 62
Item-total Statistics
Scale Scale Corrected
Mean Variance Item- Alpha
if Item if Item Total if Item
Deleted Deleted Correlation Deleted
MATHS1 25.2749 25.5752 .6614 .8629
MATHS2 25.0333 26.5322 .6235 .8661
MATHS3 25.0192 30.5174 .0996 .9021
MATHS4 24.9786 25.8671 .7255 .8589
MATHS5 25.4664 25.6455 .6707 .8622
MATHS6 25.0813 24.9830 .7114 .8587
MATHS7 25.0909 26.4215 .6208 .8662
MATHS8 25.8699 25.7345 .6513 .8637
MATHS9 25.0340 26.1201 .6762 .8623
MATHS10 25.4642 25.7578 .6495 .8638
Reliability Coefficients
N of Cases = 1353.0 N of Items = 10
Alpha = .8790
SPSS: Reliability output
Remove this item.
Maths3 does not correlate well with the
other items and the Cronbach's alpha
would increase without this item.
63. 63
Item-total Statistics
Scale Scale Corrected
Mean Variance Item- Alpha
if Item if Item Total if Item
Deleted Deleted Correlation Deleted
MATHS1 22.2694 24.0699 .6821 .8907
MATHS2 22.0280 25.2710 .6078 .8961
MATHS4 21.9727 24.4372 .7365 .8871
MATHS5 22.4605 24.2235 .6801 .8909
MATHS6 22.0753 23.5423 .7255 .8873
MATHS7 22.0849 25.0777 .6166 .8955
MATHS8 22.8642 24.3449 .6562 .8927
MATHS9 22.0280 24.5812 .7015 .8895
MATHS10 22.4590 24.3859 .6524 .8930
Reliability Coefficients
N of Cases = 1355.0 N of Items = 9
Alpha = .9024
SPSS: Reliability output
Alpha improves.
64.
65. Validity
Validity is the extent to
which an instrument actually
measures what it purports
to measure.
Validity = does the
test measure what its
meant to measure?
66. 66
Validity
• Validity is multifaceted and includes:
– Comparing wording of the items
with theory and expert opinion
– Examining correlations with similar
and dissimilar measures
– Testing how well the measure
predicts the future
68. 68
Face validity
(low-level of importance overall)
• Asks:
"At face-value, do the questions
appear to measure what the test
purports to measure?"
• Important for:
Respondent buy-in
• How assessed:
Read the test items
69. 69
Content validity
(next level of importance)
• Asks:
"Are questions measuring the
complete construct?"
• Important for:
Ensuring holistic assessment
• How assessed:
Diverse means of item generation
(lit. review, theory, interviews,
expert review)
70. 70
Criterion validity
(high importance)
• Asks:
"Can a test score predict real
world outcomes?"
• Important for:
Test relevance and usefulness
• How assessed:
Concurrent validity: Correlate test scores with
recognised external criteria such as performance appraisal scores
Predictive validity: Correlate test scores with
future outcome e.g., offender risk rating with recidivism
71. 71
Construct validity
(high importance)
• Asks:
Does the test assess the construct it
purports to?
("the truth, the whole truth and nothing but the truth.")
• Important for:
Making inferences from operationalisations
to theoretical constructs
• How assessed:
- Theoretical (is the theory about the construct valid?)
- Statistical
Convergent – correlation with similar measures
Discriminant – not correlated with other constructs
74. 74
Composite scores
Combine item-scores into an
overall factor score which
represents individual differences in
each of the target constructs.
The new composite score can then
be used for:
• Descriptive statistics and histograms
• Correlations
• As IVs and/or DVs in inferential analyses
such as MLR and ANOVA
76. 76
Unit weighting
Average (or total) of item scores
within a factor.
(each variable is equally weighted)
X = mean(y1…yp) Unit
Weighting
.25
.25 .25
.25
77. 77
Creating composite scores:
Dealing with missing data
To maximise the sample
size, consider computing
composite scores in a
way that allows for some
missing data.
78. 78
SPSS syntax:
Compute X = mean (v1, v2, v3, v4, v5, v6)
Compute X = mean.4 (v1, v2, v3, v4, v5, v6)
Specifies a min. # of items. If the min. isn't
available, the composite score will be missing.
In this example, X will be computed for a case when the case has responses to at
least 4 of the 6 items.
How many items can be missed? Depends on
overall reliability. A rule of thumb:
● Allow 1 missing per 4 to 5 items
● Allow 2 missing per 6 to 8 items
● Allow 3+ missing per 9+ items
Composite scores:
Missing data
A researcher may decide to be more
or less conservative depending on
the factors’ reliability, sample size,
and the nature of the study.
79. 79
Regression weighting
Factor score regression weighting
The contribution of each
item to the composite score
is weighted to reflect
responses to some items
more than other items.
X = .20*a + .19*b + .27*c + .34*d
X
.20
.19 .27
.34
a
b c
d
This is arguably more valid, but the
advantage may be marginal, and it
makes factor scores between
studies more difficult to compare.
80. 80
Regression weighting
Two calculation methods:
• Manual (use Compute – New variable
name = MEAN.*(list of variable names
separated by commas)
• Automatic (use Factor Analysis –
Factor Scores – Save as variables -
Regression)
81. Regression weighting – SPSS data
Data view: Data are standardised, centred around 0
Variable view: Variables auto-calculated through
SPSS factor analysis
83. 83
Writing up instrument development
• Introduction
–Review previous literature about the
construct's underlying factors – consider
both theory and research
–Generate a research question e.g., “What
are the underlying factors of X?” or make
a hypothesis about the number of factors
and what they will represent.
84. 84
Writing up instrument development
• Method
–Materials/Instrumentation – summarise the
design and development of the measures
and the expected factor structure
e.g., present a table of the expected factors and
their operational definitions.
85. 85
Writing up instrument development
• Results
–Factor analysis
• Assumption testing
• Extraction method & rotation
• # of factors, with names and definitions
• # of items removed and rationale
• Item factor loadings & communalities
• Factor correlations
–Reliability for each factor
–Composite scores for each factor
–Correlations between factors
86. 86
Writing up instrument development
• Discussion
–Theoretical underpinning – Was it
supported by the data? What adaptations
should be made to the theory?
–Quality / usefulness of measure – Provide
an objective, critical assessment,
reflecting the measures' strengths and
weaknesses
–Recommendations for further improvement
• Writing up a factor analysis
–Download examples: http://goo.gl/fD2qby
88. 88
Summary:
What is psychometrics?
1. Science of psychological
measurement
2. Goal: Validly measure individual
psychosocial differences
3. Design and test psychological
measures e.g., using
1. Factor analysis
2. Reliability and validity
89. 89
Summary:
Concepts & their measurement
1. Concepts name common elements
2. Hypotheses identify relations between
concepts
3. Brainstorm indicators of a concept
4. Define the concept
5. Draft measurement items
6. Pre-test and pilot test
7. Examine psychometric properties
8. Redraft/refine and re-test
91. 91
Summary: Reliability
1. Consistency or reproducibility
2. Types
1. Internal consistency
2. Test-retest reliability
3. Rule of thumb
1. > .6 OK
2. > .8 Very good
4. Internal consistency
1. Split-half
2. Odd-even
3. Cronbach's Alpha
92. 92
Summary: Validity
1. Extent to which a measure measures
what it is intended to measure
2. Multifaceted
1. Compare with theory and expert opinion
2. Correlations with similar and dissimilar
measures
3. Predicts future
93. 93
Summary: Composite scores
Ways of creating composite (factor)
scores:
1. Unit weighting
1.Total of items or
2. Average of items
(recommended for lab report)
2. Regression weighting
1. Each item is weighted by its
importance to measuring the underlying
factor (based on regression weights)
94. 94
1. Introduction
1. Review constructs & previous structures
2. Generate research question or hypothesis
2. Method
1. Explain measures and their development
3. Results
1. Factor analysis
2. Reliability of factors
3. Descriptive statistics for composite scores
4. Correlations between factors
4. Discussion
1. Theory? / Measure? / Recommendations?
Summary: Writing up
instrument development
95. 95
1. Allen, P. & Bennett, K. (2008). Reliability analysis (Ch 15) in SPSS
for the health & behavioural sciences (pp. 205-218). South
Melbourne, Victoria, Australia: Thomson.
2. Bryman, A. & Cramer, D. (1997). Concepts and their measurement
(Ch. 4). In Quantitative data analysis with SPSS for Windows: A
guide for social scientists (pp. 53-68). Routledge.
3. DeCoster, J. (2000). Scale construction notes.
http://www.stat-help.com/scale.pdf (pdf)
4. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J.
(1999). Evaluating the use of exploratory factor analysis in
psychological research. Psychological Methods, 4(3), 272-299.
5. Fowler, F. (2002). Designing questions to be good measures. In
Survey research methods (3rd ed.)(pp. 76-103). Thousand Oaks,
CA: Sage. Ereserve.
6. Howitt, D. & Cramer, D. (2005). Reliability and validity: Evaluating
the value of tests and measures (Ch. 13). In Introduction to
research methods in psychology (pp. 218-231). Harlow, Essex:
Pearson. eReserve.
References
96. 96
Open Office Impress
● This presentation was made using
Open Office Impress.
● Free and open source software.
● http://www.openoffice.org/product/impress.html
Editor's Notes
7126/6667 Survey Research & Design in Psychology
Semester 1, 2017, University of Canberra, ACT, Australia
James T. Neill
Home page: http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology
Lecture page: http://en.wikiversity.org/wiki/Psychometric_instrument_development
Image source:http://commons.wikimedia.org/wiki/File:Soft_ruler.jpg
Image author: Lite, http://commons.wikimedia.org/wiki/User:Lite
Image license: GFDL 1.2+, CC-by-SA 3.0 unported
Description:
This lecture recaps the previous lecture on exploratory factor analysis, and introduces psychometrics and (fuzzy) concepts and their measurement, including (operationalisation), reliability (particularly internal consistency of multi-item measures), validity and the creation of composite scores.
This lecture is accompanied by a computer-lab based tutorial.
http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Tutorials/Psychometrics
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Image source:https://commons.wikimedia.org/wiki/File:Pleiades_large.jpg
Image authorr: NASA, ESA, AURA/Caltech, Palomar Observatory
The science team consists of: D. Soderblom and E. Nelan (STScI), F. Benedict and B. Arthur (U. Texas), and B. Jones (Lick Obs.)
License: https://en.wikipedia.org/wiki/Public_domain
Image source: https://www.flickr.com/photos/lenore-m/441579944/in/set-72157600039854497
Image author: Lenore Edman
Image license: CC by 2.0, https://creativecommons.org/licenses/by/2.0/
Bryman & Cramer “Concepts & their Measurement” (1997):&quot;Concepts form a linchpin in the process of social research. Hypotheses contain concepts which are the products of our reflections on the world. Concepts express common elements in the world to which we give a name.&quot; (p. 53)
Bryman & Cramer “Concepts & their Measurement” (1997):
“Once formulated, a concept and the concepts with which it is purportedly associated, such as social class and authoritarianism, will need to be operationally defined, in order for systematic research to be conducted in relation to it...&quot;
Bryman & Cramer “Concepts & their Measurement” (1997):
“Once formulated, a concept and the concepts with which it is purportedly associated, such as social class and authoritarianism, will need to be operationally defined, in order for systematic research to be conducted in relation to it...&quot;
Bryman & Cramer “Concepts & their Measurement” (1997, p. ?)
Image (left) source:http://www.flickr.com/photos/iliahi/408971482/
By Maui in Vermont: http://www.flickr.com/photos/iliahi
License: CC-by-A 2.0
Image (right) source: http://commons.wikimedia.org/wiki/File:3DTomo.jpg
By Tomograph: http://de.wikipedia.org/wiki/Benutzer:Tomograph
License: GFDL
Image source: James Neill, Creative Commons Attribution 3.0; Adapted from original image source: http://www.uwo.ca/fhs/nursing/laschinger/image002.gif (Retreived 20 March, 2007). As of 25 March, 2008, the original image could no longer be found. It was based on research by Laschinger and colleagues on nurse empowerment.
&lt;number&gt;
Image source: Figure 4.2 Bryman & Cramer (1997)
Image source: http://www.freestockphotos.biz/stockphoto/17079
Image author: National Cancer Institute, https://visualsonline.cancer.gov/
Image license: Public domain
&lt;number&gt;
Image source:s Unknown.
Paradigm (e.g., assumptions, focus, collection method)
Personal researcher bias (conscious & unconscious)
Sampling (e.g., representativeness of sample)
Non-sampling (e.g, non-response, inaccurate response due to unreliable measurements, misunderstanding, social desirability, faking bad, researcher expectancy, administrative error)
“The lower the precision, the more subjects you&apos;ll need in your study to make up for the &quot;noise&quot; in your measurements. Even with a larger sample, noisy data can be hard to interpret. And if you are an applied scientist in the business of testing and assessing clients, you need special care when interpreting results of noisy tests.”
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
Image source: http://www.flickr.com/photos/ajawin/2614485186/
By akawin http://www.flickr.com/photos/ajawin/
License: Creative Commons by Attribution 2.0 http://creativecommons.org/licenses/by/2.0/deed.en
&lt;number&gt;
Psychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the study of differences between individuals and between groups of individuals.
http://en.wikipedia.org/wiki/Psychometrics
&lt;number&gt;
Psychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the study of differences between individuals and between groups of individuals.
http://en.wikipedia.org/wiki/Psychometrics
&lt;number&gt;
http://www.nytimes.com/2006/05/05/education/05testers.html
In Australia, probably the largest collection of psychometricians is at the Australian Council for Educational Research:
http://www.acer.edu.au/
&lt;number&gt;
Also item response modeling
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg
License: Public domain
Image source: http://www.flickr.com/photos/psd/17433783/in/photostream
By psd http://www.flickr.com/photos/psd/
License: Creative Commons by Attribution 2.0 http://creativecommons.org/licenses/by/2.0/deed.en
Internal consistency (single administration) and Test-reliability (multiple administrations) are based on classical test theory. A more advanced/refined reliability technique is based Item Response Theory, which involves examining the extent to which each item discriminates between individuals with high/low overall scores.
For more info see: http://en.wikipedia.org/wiki/Reliability_(psychometric)
Image source: http://www.socialresearchmethods.net/kb/relandval.php
Reliability is about the consistency of a measure.
Validity is about whether a test actually measures what its meant to measure.
Validity requires reliability, but reliability alone is not sufficient for validity.
See the more complex discussion of this at:
http://en.wikiversity.org/wiki/Reliability_and_validity
Internal consistency (single administration) and Test-reliability (multiple administrations) are based on classical test theory. A more advanced/refined reliability technique is based Item Response Theory, which involves examining the extent to which each item discriminates between individuals with high/low overall scores.
For more info see: http://en.wikipedia.org/wiki/Reliability_(psychometric)
Image source: Unknown
Reliable; not valid
Reliable; valid
Internal consistency (single administration) and Test-reliability (multiple administrations) are based on classical test theory. A more advanced/refined reliability technique is based Item Response Theory, which involves examining the extent to which each item discriminates between individuals with high/low overall scores.
For more info see: http://en.wikipedia.org/wiki/Reliability_(psychometric)
Image source: Table 7 Fabrigar et al (1999).
Reference
Howitt & Cramer (2005), p. 221
Reference
Howitt & Cramer (2005), p. 221
Reference
Howitt & Cramer (2005), p. 221
&lt;number&gt;
This example is from Francis 6.1
This example is from Francis 6.1
This example is from Francis 6.1
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
SPSS – Analyze – Scale – Reliability Analysis
This example is from Francis 6.1
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
SPSS – Analyze – Scale – Reliability Analysis
This example is from Francis 6.1
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
SPSS – Analyze – Scale – Reliability Analysis
This example is from Francis 6.1
This example is from Francis 6.1
This example is from Francis 6.1
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
Life Effectiveness Questionnaire - http://wilderdom.com/leq.html
For the lab report, students won&apos;t have test-retest reliability, but they can include a column that indicates the number of items.
From http://www.socialresearchmethods.net/kb/measval.htm
From http://www.socialresearchmethods.net/kb/measval.htm
Adapted from: http://www.psyasia.com/supportsuite/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=85&nav=0
Prima facie extent to which an item is judged to reflect target construct.
e.g., a test of ability to add two-digit numbers should cover the full range of combinations of digits. A test with only one-digit numbers, or only even numbers, would (on the face of it) not have good coverage of the content domain.
Adapted from: http://www.psyasia.com/supportsuite/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=85&nav=0
Extent to which test content covers a representative sample of the domain to be measured. Compare test with:
existing literature
expert panels
qualitative interviews / focus groups with target sample
Adapted from: http://www.psyasia.com/supportsuite/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=85&nav=0
Involves the correlation between the test and a criterion variable (or variables) taken as representative of the construct.
Concurrent validity
Correlation between the measure and other recognised measures of the target construct
Predictive validity
Extent to which a measure predicts something that it theoretically should be able to predict.
Criterion refers to the “true” score. Criterion validity refer to measures of relationship between the criterion variable and the measured variable.
http://en.wikipedia.org/wiki/Concurrent_validity
http://www.socialresearchmethods.net/kb/measval.htm
Adapted from: http://www.psyasia.com/supportsuite/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=85&nav=0
Totality of evidence about whether a particular operationalisation of a construct adequately represents what is intended.
Not distinct from support for the substantive theory of the construct.
Involves empirical and theoretical support.
Includes:
Statistical analyses of the internal structure of the test including the relationships between responses to different test items (internal consistency).
Relationships between the test and measures of other constructs.
Convergent validity
Extent to which a measure correlates with measures with which it theoretically should be associated.
Discriminant validity
Extent to which a measure does not correlate with measures with which it theoretically should not be associated.
Image source: https://commons.wikimedia.org/wiki/File:PEO-hamburger.svg
Image author: PhantomOpenEmoji, https://github.com/break24/PhantomOpenEmoji/graphs/contributors
Image license: CC by 3.0 unported, https://creativecommons.org/licenses/by/3.0/deed.en
&lt;number&gt;
&lt;number&gt;
&lt;number&gt;
&lt;number&gt;
compute X = mean (v1, v2, v3, v4, v5.v6)(note: a missing value will be returned for a case if 1 or more of its items are missing data)
compute X = mean.4 (v1, v2, v3, v4, v5,v6) (note: in this case a missing value will be returned for a case if 3 or more of its items are missing data – the “.4” means calculate a mean using data from 4 or more items, otherwise make X missing)
X = r1y1 +…. + rpyp
Regression weighting will also include small weights for non-target items