2. Operationalization Choices
• Conceptualization is the refinement
and specification of abstract
concepts
• Then we operationalize (developing rules
for inclusion and exclusion) based on
previous definitions or based on concepts
in conceptual definition
3. Measurement
• The assignment of numerals to
objects or events according to rules
4. Empiricism
• What to observe
• Whom to observe
• How to observe
• When to observe
• How to analyze the data
5. Indicators and Dimensions
• Indicator (variable) – An observation
that we choose to consider as a
reflection of a variable we wish to
study.
• Dimension – A specifiable aspect of
a concept.
6. Stevens
• Distinguished 4 types of “measurement
scales”
– Nominal
– Ordinal
– Interval
– Ratio
• Theoretical structures are represented by
data structures
– Data helps to focus theory
7. Levels of Measurement – Nominal
• Variables who attributes have only
the characteristics of exhaustiveness
and mutually exclusiveness
– Examples: gender, religious affiliation,
college major, hair color, birthplace,
nationality
8. Levels of Measurement – Ordinal
• Variables with attributes we can
logically rank order
– Examples: socioeconomic status, level
of conflict, prejudice, conservativeness,
ranking of topics,
9. Levels of Measurement – Interval
• Variables for which the actual
distance between attributes has
meaning.
– Temperature, IQ score
10. Levels of Measurement – Ratio
• Variables whose attributes meet the
requirements of a interval measure,
and has a true zero point.
–Age, length of time, number of
organizations, number of groups
11.
12. Judge the relative success (or failure) in
measuring various concepts
• Reliability (consistency of measurement)
• Validity (confidence in measures)
13. Reliability
• Consistency of measurement
– Reproducibility over time
– Consistency between different coders
– Consistency among indicators
14. Measurement Validity
• Are we really measuring the concept we
defined?
– Is it a valid way to measure the concept?
• Many approaches to validation
15. Key to Reliability and Validity
• Concept explication
• Conceptual definition
• Operational definition
16. Four Aspects of Reliability
1. Stability
2. Reproducibility
3. Homogeneity
4. Accuracy
17. 1. Stability
• Consistency across time
– Repeating a measure at a later time to
measure consistency
– Compare Time 1 and Time 2
18. 2. Reproducibility
• Consistency between observers
• Equivalent application of measuring
device
– Do observers reach the same conclusion?
19. 3. Homogeneity
• Consistency between different measures
of the same concept
20. Alpha – Self-efficacy
• Even when things are tough, I can perform
quite well.
• I will be able to successfully overcome
many challenges.
• I don’t achieve most goals that I have set
for myself.
• I like ice cream.
21. 4. Accuracy
• Mistakes in measurement
• Observers must have sufficient:
– Training
– Motivation
– Concentration
22. Increasing Reliability
• General
– Training coders/interviewers/lab personnel
– More careful concept explication
• Survey
– Increase number of items on scale
– Weed out bad items
• Content Analysis
– Improve definitions of content categories
– Remove bad coders
23. 1. Face Validity
• Subjective evaluation from experts
• Compare items to conceptual definition
24. 2. Content Validity
(Face Validity)
• Subjective agreement about what is not there
• Start with conceptual definition of each dimension
– It is operated at an operational definition
– Over- or under-represented
• Example: Civic participation
– Did you vote in the last election?
– Do you belong to civic groups?
– Have you ever attended a city council meeting?
25. 3. Predictive validity (criterion)
• The extent to which one measure predicts
the values of another measure
26. 4. Convergent Validity
• Measuring a concept with different
methods
– If different methods yield the same results,
convergent validity is supported
Editor's Notes
Two levels of investigations: conceptual and empirical. Representational theory is about providing concrete structure to our theory. What are the features that we can observe to translate concepts into reality. When you represents something, there are intentions. We assign numbers to properties. For example, numbers can represent the length of physical objects. Numbers give the object greater meaning.
Since the early 1900s, social scientists have been trying to measure reality. We measure so we can manipulate the universe. We translate our observations into numerical form, but our measures have to measure what they intend to measure
Identifying the concepts in the conceptual definitions is key to conceptualizing
Observations are checked against data
Operationalization is the development of specific research procedures that will result in empirical observations representing those concepts in the real world.
Remember in any conceptual definition, we want to highlight the concepts and think about what they mean. For concept definition exercise, I want you to look at concepts.
Stevens in the 1950s. Endorsement within social sciences. He believe any psychological state.
Objects and events that are measured, not magnitudes. Careful, deliberate observations of the real world we can describing objects and events. They are most often described in terms of the attributes…. Objects(room) and events (rooms changing color) … attributes of objects (length of a room) and attributes of events (time it takes to paint a room). His definition focuses on objects, however it is really attributes of objects that are being measured.
Numerals, not numbers. Numerals - a figure, symbol, or group could be denote a number… meaning that numerals could be assigned to anything.
Assignment (representation) and rules…. Methods are the set of rules that you learn. As we learned last class, we should be concerned with how we make those decisions. Numerical representation. Map measurement.
Making planned observations
Boredom in the Classroom
Students- elementary
Should we interview them or observe them
Friday afteroon may be a difficult time
Percentage bored? Percentage participating. Correlate boredom with other variables
Representational theory tie to “scales of measurement”… to encourage consistent rules when observing objects or events. Interval measurement is most commonly applied.
This is going to affect how operationalize a variable.
Attributes are defined by levels of variability
You must think about level of measurement when deciding… because it will influence that statistics you choose to analyze your data.
Social attitudes are at ordinal
Most people view scores on intelligence tests are ordinal because scores have not been shown to correspond to equal differences.
Intervals have equal distances but do not have an absolute zero
Validity also extends to:
Precision in the design of the study – ability to isolate causal agents while controlling other factors
(Internal Validity)
Ability to generalized from the unique and idiosyncratic settings, procedures and participants to other populations and conditions
(External Validity)
\
Statistical coefficients that tell use how consistently we measured something
Reliability – That quality of measurement method that suggests that the same data would have been collected each time in repeated observations of the same phenomenon.
Reliability is not the same as accuracy.
Judgmental as well as empirical aspects
Meaning analysis
Defining what a concept means
Spelling out how we are going to measure concept
Computer self-efficacy: I have the ability to describe how a computer works and I have the ability to install software
If we don’t get the same results, what are we measuring? Intercoder reliability. Photograph – visual presentation of self--- visual content cu of person smilling.
Lack of reliability can compromise validity
Different items used to tap a given concept show similar results – ex. open-ended and closed-ended questions
Internal consistency -
The extent to which items are highly correlated with one another – research self-efficacy
Number of dimensions influence alpha levels (Cortina, 1993) Cronbach’s Alpha Reliability
Coefficients
0.90 Excellent
higher suggests item redundancy
0.80 Very Good
0.70 Adequate (Above recommended)
.60 (exploratory research – Hair, et al., 2010)
0.50 Poor
Increased by clear, defined procedures
Reduce complications that lead to errors
Contnet
Specification of procedures/rules…Reduce subjectivity (room for interpretation)
Specification at a manifest leve;
Make judgments about superficial appearance.
Subjective judgment of experts about:
“what’s there”
Do the measures make sense?
That quality of an indicator that makes it seem a reasonable measure of some variable.
Make judgments about appropriateness of content
Content Validity – The degree to which a measure coves the range of meanings included within a concept.
Content validity refers to whether the items on your test actually test what you're looking at --- do your measures actually measure what you say that they are meausirng – factor analysis
Construct Validity – The degree to which a measure relates to other variables as expected within a system of theoretical relationships
If current indicators are insufficient: develop and add more indicators
What about “protest participation” or “online organizing”?
Operational level technique of conceptualization
Factor analysis
So lets say for example, I am looking for a doctoral student for my next research project. I ask for samples of their writing. Writing samples show some good and poor writing. I am looking at their writing as a predictor as research success.
Communication apprehensiveness – student participation (negative relationship)
Correlation between different methods measuring the same trait
