The document discusses validity and reliability in research methods. It defines different types of validity like face validity, content validity, construct validity, concurrent validity, and predictive validity. It also discusses how to assess validity through multiple indicators and dimensions. The document also covers reliability and how to increase reliability through retesting, clear question wording, and avoiding errors in data collection and coding. It describes systematic and random errors and how to minimize biases.

1. 07.10.08 POLI 399<br />Assignments will be handed back Friday via e-mail. Comments will be imbedded in ‘Track Changes’ (Control, Shift, E.) (Command , shift, E?)<br />Cont. ‘Getting Down to the Nitty Gritty’<br />Quickest way to know if you have intervals is if there are no labels attached to it.<br />Validity is one of the core concepts. It is not just the things you are measuring, but also the whole of your research project. External and Internal Validity.<br />Components Properties <br />Face Construct}<br />ContentConcurrent} Measures<br />Predictive}<br /> <br />Validity<br /> <br />External Internal} Research Project<br />Does it reflect real life? Does it make sense to the people being studies?<br />Is it generalizable?<br />Must some kind of criteria by which you judge (or measure)? There is not true or real thing. These are things that we make up. Are we measuring according to our definition? <br />Two Components for Assessing Validity:<br /> Face Validity: Exists if there is general agreement. On the face of it, does it seem like a good way to measure? Does it make sense? Is it logical? Not enough that you think so, there must be some kind of agreement among others.<br />Content Validity: Have I captured all the elements? Exists if the various dimensions are being captured.<br />This is about how you have Operationalized your data.<br />Properties for Assessing Validity:<br />Construct Validity: Is the data doing what it theoretically should be doing? If you don’t know (contested data) than there is no way to assess.<br />-If it is not doing what is should: you either measured wrong, or the theory is wrong (times could have changed)<br />-Be very careful about assuming that you have done things correctly. 9 times out of 10, you have made an error and not done what you thought you have with the data.<br />Concurrent Validity: If you have different measures to measure similar things, they should be related. If they are not related, there could be a problem. They should be relatively similar if they were tested at the same time. (Two different measures of the same concept should act in the same way.)<br />Predictive Validity: Exists if it can predict an outcome at a later date according to expectations.<br />Refer to whether the properties of a measure act in the way you think it should, at least statistically speaking.<br />Validity is a quality. Measures will be more or less valid. (Not “valid” or “Invalid”). Must justify. It is not enough just to say it is.<br />If you can use multiple indicators, you are head of the game. If you just use one, it is harder to determine validity. Two or three can be measured against one another.<br />-Social Science is multi-dimensional, it therefore makes sense that you should measure more than one indicator. SES is complex!<br />Reliability: How close are the arrows clustered around the bulls eye?<br />Consistency, Stability. Will your answers be the same from day to day? In a survey question, you increase reliability by checking your wording. <br />-The minute you make people guess, you will have errors in your data. <br />-It is also better to ask people about themselves, than about friends or family.<br />-There can also be social pressure to respond a certain way.<br />Increasing Reliability:<br />Can retest and check for consistency<br />Can check an external source<br />This adds complexity, but it also can be impossible due to privacy laws.<br />Make sure your questions are clear, and that respondents can actually answer the question. Don’t assume a knowledge base that may not be there. Don’t use colloquial/slang terms. Am I using language that will be understood? (Must have a Post Grad degree to understand the a university calendar!) E.g. Men will guess, but women more likely to say ‘don’t know’.<br />Errors can be introduced on the part of whoever is coding the data. Every time you transfer data, error can be introduced. Should always double check!<br />Measurement error: Any time your measures deviate from the “true” value.<br />The less reliable the measure, the more measurement error you are likely to have.<br />If it is not a valid measure, you will have less reliability.<br />If it is a more valid measure, you will have more reliability.<br />It also matters who is doing the interviewing. Women more likely to answer sensitive questions to other women. This is an important consideration. Men more likely to make something up.<br />Measure = TV + SE + RE<br />__________<br />(value)<br />TV=True Value<br />SE=Systematic Error (want to minimize this as much as possible)<br />RE=Random Error (also called “noise”)<br />Random Error: differs between cases. Some are high, some are low. But there is not a pattern to the error. The minute you introduce recall you have random error. Guessing is also random error.<br />This is the kind of error that you do not have to worry about too much with a large sample.<br />Systematic Error: (Bias.) This is persistent error in a particular direction. (Either always high or always low.)<br />-Through wording or social pressure<br />-It is a concern if you want to say something about overall average.<br />Avoiding Errors:<br />-Take several measures, and average out<br />-Can also use multiple indicators<br />-Employ random sampling (control or sample, no systematic bias in how you sample)<br />-Survey, computer randomly generates numbers.<br />-Use sensitive measures (multiple categories rather than yes/no or agree/disagree)<br />-Avoid confusing wording and instructions (k.i.s.s. rule: keep it simple and straight-forward)<br />-Check data for errors (Run multiple frequencies in SPSS)<br />Indexes and Scales combine responses and several measures.<br />-Better capture the multiple dimensions of a concepts<br />-Include parsimony: more information, and better data in a simple way<br />Must think about how you are putting your indexes together.<br />-Each variable should be discriminating (including two of the same kinds of questions is a waste. E.g. 7 questions on abortion). Each should be slightly different.<br />-Must check which variables you can use CCS Code Book<br />-Correlations – move together in a consistent analysis<br />-Cronbach’s Alpha – gives an overall number which tells you how closely related these are<br />-The more variables, the higher the number. Books says .7, but this is too high for most purposes. Range is from 0-1, but you never get 1. .4 is the number we will use for this class, but this would be too low for a grad thesis.<br />-If you recode, you must always use the recoded variable.<br />