Forms of Reliability and Validity in statistics

1. Forms of Reliability
and Validity

2. In everyday life, we probably use reliability to describe how something is valid. However, in
research and testing, reliability and validity are not the same things.
When it comes to data analysis, reliability refers to how easily replicable an outcome is. For
example, if you measure a cup of rice three times, and you get the same result each time, that result is
reliable.
The validity, on the other hand, refers to the measurement’s accuracy. This means that if the
standard weight for a cup of rice is 5 grams, and you measure a cup of rice, it should be 5 grams.
So, while reliability and validity are intertwined, they are not synonymous. If one of the
measurement parameters, such as your scale, is distorted, the results will be consistent but invalid.
Data must be consistent and accurate to be used to draw useful conclusions.

3. What is Reliability?
When a measurement is consistent it’s reliable. But of course, reliability
doesn’t mean your outcome will be the same, it just means it will be in the same
range.
For example, if you scored 95% on a test the first time and the next you score,
96%, your results are reliable. So, even if there is a minor difference in the outcomes,
as long as it is within the error margin, your results are reliable.
Reliability allows you to assess the degree of consistency in your results. So, if
you’re getting similar results, reliability provides an answer to the question of how
similar your results are.

4. What is Validity?
A measurement or test is valid when it correlates with the expected
result. It examines the accuracy of your result.
To establish the validity of a test, the results must be consistent. Looking
at most experiments (especially physical measurements), the standard value
that establishes the accuracy of a measurement is the outcome of repeating the
test to obtain a consistent result.

5. For example, before I can conclude that all 12-inch rulers are one foot, I must repeat the
experiment several times and obtain very similar results, indicating that 12-inch rulers are indeed one
foot.
Most scientific experiments are inextricably linked in terms of validity and reliability. For example,
if you’re measuring distance or depth, valid answers are likely to be reliable.
But for social experiences, one isn’t the indication of the other. For example, most people believe
that people that wear glasses are smart.
Of course, we’ll find examples of people who wear glasses and have high IQs (reliability), but the truth is
that most people who wear glasses simply need their vision to be better (validity).

6. Assessing Reliability
When assessing reliability, we want to know if the measurement can be
replicated. We’d have to change some variables to ensure that this test holds,
the most important of which are time, items, and observers.
If the main factor you change when performing a reliability test is time,
you’re performing a test-retest reliability assessment.

7. Assessing Reliability
However, if you are changing items, you are performing an internal
consistency assessment. It means you’re measuring multiple items with a
single instrument.
If you’re measuring the same item with the same instrument but using
different observers or judges, you’re performing an inter-rater reliability test.

8. Assessing Validity
Evaluating validity can be more tedious than reliability. With reliability,
you’re attempting to demonstrate that your results are consistent, whereas, with
validity, you want to prove the correctness of your outcome.
Although validity is mainly categorized under two sections (internal and
external), there are more than fifteen ways to check the validity of a test. In this
article, we’ll be covering four.

9. Content Validity
It measures whether the test covers all the content it needs to provide
the outcome you’re expecting.
Suppose I wanted to test the hypothesis that 90% of Generation Z uses
social media polls for surveys while 90% of millennials use forms. I’d need a
sample size that accounts for how Gen Z and millennials gather information.

10. Criterion Validity
Criterion validity is when you compare your results to what you’re
supposed to get based on a chosen criteria. There are two ways these could be
measured, predictive or concurrent validity.
Face Validity
It’s how we anticipate a test to be. For instance, when answering a
customer service survey, I’d expect to be asked about how I feel about the
service provided.

11. Construct-Related Validity
This is a little more complicated, but it helps to show how the validity of
research is based on different findings.
As a result, it provides information that either proves or disproves that
certain things are related.

12. Types of Reliability
We have three main types of reliability assessment and here’s how they work:

13. Test-retest Reliability
This assessment refers to the consistency of outcomes over time. Testing reliability over time
does not imply changing the amount of time it takes to conduct an experiment; rather, it means repeating
the experiment multiple times in a short time.
For example, if I measure the length of my hair today, and tomorrow, I’ll most likely get the same
result each time.
A short period is relative in terms of reliability; two days for measuring hair length is considered
short. But that’s far too long to test how quickly water dries on the sand.
A test-retest correlation is used to compare the consistency of your results. This is typically a
scatter plot that shows how similar your values are between the two experiments.
If your answers are reliable, your scatter plots will most likely have a lot of overlapping points,
but if they aren’t, the points (values) will be spread across the graph.

14. Internal Consistency
It’s also known as internal reliability. It refers to the consistency of results for various items when measured on the same scale.
This is particularly important in social science research, such as surveys, because it helps determine the consistency of people’s
responses when asked the same questions.
Most introverts, for example, would say they enjoy spending time alone and having few friends. However, if some introverts
claim that they either do not want time alone or prefer to be surrounded by many friends, it doesn’t add up.
These people who claim to be introverts or one this factor isn’t a reliable way of measuring introversion.
Internal reliability helps you prove the consistency of a test by varying factors. It’s a little tough to measure quantitatively but you
could use the split-half correlation.
The split-half correlation simply means dividing the factors used to measure the underlying construct into two and plotting them
against each other in the form of a scatter plot.
Introverts, for example, are assessed on their need for alone time as well as their desire to have as few friends as possible. If
this plot is dispersed, likely, one of the traits does not indicate introversion.

15. Inter-Rater Reliability
This method of measuring reliability helps prevent personal bias. Inter-rater reliability assessment helps judge
outcomes from the different perspectives of multiple observers.
A good example is if you ordered a meal and found it delicious. You could be biased in your judgment for several
reasons, perception of the meal, your mood, and so on.
But it’s highly unlikely that six more people would agree that the meal is delicious if it isn’t. Another factor that could
lead to bias is expertise. Professional dancers, for example, would perceive dance moves differently than non-professionals.
So, if a person dances and records it, and both groups (professional and unprofessional dancers) rate the video,
there is a high likelihood of a significant difference in their ratings.
But if they both agree that the person is a great dancer, despite their opposing viewpoints, the person is likely a
great dancer.

16. Types Validity
Researchers use validity to determine whether a measurement is
accurate or not. The accuracy of measurement is usually determined by
comparing it to the standard value.
When a measurement is consistent over time and has high internal
consistency, it increases the likelihood that it is valid.

17. Content Validity
This refers to determining validity by evaluating what is being measured. So content validity
tests if your research is measuring everything it should to produce an accurate result.
For example, if I were to measure what causes hair loss in women. I’d have to consider things
like postpartum hair loss, alopecia, hair manipulation, dryness, and so on.
By omitting any of these critical factors, you risk significantly reducing the validity of your
research because you won’t be covering everything necessary to make an accurate deduction.
For example, a certain woman is losing her hair due to postpartum hair loss, excessive
manipulation, and dryness, but in my research, I only look at postpartum hair loss. My research will show
that she has postpartum hair loss, which isn’t accurate.
Yes, my conclusion is correct, but it does not fully account for the reasons why this woman is
losing her hair.

18. Criterion Validity
This measures how well your measurement correlates with the variables you want
to compare it with to get your result. The two main classes of criterion validity are
predictive and concurrent.

19. Predictive Validity
It helps predict future outcomes based on the data you have. For example, if a
large number of students performed exceptionally well in a test, you can use this to
predict that they understood the concept on which the test was based and will perform
well in their exams.

20. Concurrent Validity
On the other hand, involves testing with different variables at the same time. For
example, setting up a literature test for your students on two different books and
assessing them at the same time.
You’re measuring your students’ literature proficiency with these two books. If your
students truly understood the subject, they should be able to correctly answer questions
about both books.

21. Face Validity
Quantifying face validity might be a bit difficult because you are measuring the perception
validity, not the validity itself. So, face validity is concerned with whether the method used for
measurement will produce accurate results rather than the measurement itself.
If the method used for measurement doesn’t appear to test the accuracy of a
measurement, its face validity is low.
Here’s an example: less than 40% of men over the age of 20 in Texas, USA, are at least
6 feet tall. The most logical approach would be to collect height data from men over the age of
twenty in Texas, USA.
However, asking men over the age of 20 what their favorite meal is to determine their
height is pretty bizarre. The method I am using to assess the validity of my research is quite
questionable because it lacks correlation to what I want to measure.

22. Construct-Related Validity
Construct-related validity assesses the accuracy of your research by collecting
multiple pieces of evidence. It helps determine the validity of your results by comparing
them to evidence that supports or refutes your measurement.

23. Convergent Validity
If you’re assessing evidence that strongly correlates with the concept, that’s
convergent validity.

24. Discriminant Validity
Examines the validity of your research by determining what not to base it on. You
are removing elements that are not a strong factor to help validate your research. Being a
vegan, for example, does not imply that you are allergic to meat.

25. Discriminant Validity
Examines the validity of your research by determining what not to base it on. You
are removing elements that are not a strong factor to help validate your research. Being a
vegan, for example, does not imply that you are allergic to meat.

26. Ensuring Reliability
To enhance the reliability of your research, you need to apply your measurement method
consistently. The chances of reproducing the same results for a test are higher when you
maintain the method you’re using to experiment.
For example, you want to determine the reliability of the weight of a bag of chips using a
scale. You have to consistently use this scale to measure the bag of chips each time you
experiment.
You must also keep the conditions of your research consistent. For instance, if you’re
experimenting to see how quickly water dries on sand, you need to consider all of the weather
elements that day.
So, if you experimented on a sunny day, the next experiment should also be conducted
on a sunny day to obtain a reliable result.

27. Ensuring Validity
There are several ways to determine the validity of your research, and the majority
of them require the use of highly specific and high-quality measurement methods.
Before you begin your test, choose the best method for producing the desired
results. This method should be pre-existing and proven.
Also, your sample should be very specific. If you’re collecting data on how dogs
respond to fear, your results are more likely to be valid if you base them on a specific
breed of dog rather than dogs in general.