TAMING THE RESEARCH BEAST Dr. Shamanthakamani Narendran M.D.(Pead), Ph.D. (Yoga Science)
INTRODUCTION How do you feel when you are assigned a research project? Do research projects frighten you ? This module is designed to help you overcome your "research fright" by... ...helping you to understand what research is and why it is important, and... ...showing you how to go about planning your own research projects, and... ...getting you started on the actual writing of the research project you have planned.
GOALS By the time you're finished with this module, you should be able to... <ul><li>Understand and interpret research reports when you read them, indentifying their major elements and underlying methodologies. </li></ul><ul><li>Plan and carry out your own research projects in the field of TESL and language acquisition. </li></ul>
In addition... <ul><li>This module is intended to supplement what you will learn from your coursework and reading. It may also help you get an idea of what research is before you take courses in research methods. </li></ul><ul><li>This module will also introduce you to basic terminology and concepts common to the research typically done by TESL and language acquisition experts. </li></ul>
Definition of Research In order to plan and carry out research, it is necessary for you to know what we mean by research-in general, as well as in the specialized fields of language teaching and language acquisition. <ul><li>Research is an </li></ul><ul><li>ORGANIZED and </li></ul><ul><li>SYSTEMATIC way of </li></ul><ul><li>FINDING ANSWERS to </li></ul><ul><li>QUESTIONS . </li></ul>
SYSTEMATIC because there is a definite set of procedures and steps which you will follow. There are certain things in the research process which are always done in order to get the most accurate results. ORGANIZED in that there is a structure or method in going about doing research. It is a planned procedure, not a spontaneous one. It is focused and limited to a specific scope.
FINDING ANSWERS is the end of all research. Whether it is the answer to a hypothesis or even a simple question, research is successful when we find answers. Sometimes the answer is no, but it is still an answer. QUESTIONS are central to research. If there is no question, then the answer is of no use. Research is focused on relevant, useful, and important questions. Without a question, research has no focus, drive, or purpose.
EPISTEMOLOGY First of all, you should realize that research is only one of several ways of "knowing." The branch of philosophy that deals with this subject is called EPISTEMOLOGY .
Epistemologists generally recognize at least four different sources of knowledge: INTUITIVE KNOWLEDGE takes forms such as belief, faith, intuition, etc. It is based on feelings rather than hard, cold "facts." AUTHORITATIVE KNOWLEDGE is based on information received from people, books, a supreme being, etc. Its strength depends on the strength of these sources.
LOGICAL KNOWLEDGE is arrived at by reasoning from "point A" (which is generally accepted) to "point B" (the new knowledge). EMPIRICAL KNOWLEDGE is based on demonstrable, objective facts (which are determined through observation and/or experimentation).
Research often makes use of all four of these ways of knowing: INTUITIVE (when coming up with an initial idea for research) AUTHORITATIVE (when reviewing the professional literature) LOGICAL (when reasoning from findings to conclusions) EMPIRICAL (when engaging in procedures that lead to these findings) Nevertheless, this last kind of knowledge, empirical knowledge, is what most modern research in TESL and language acquisition aims at establishing. That is why we call it empirical research .
EMPIRICAL RESEARCH A common image of "research" is a person in a laboratory wearing a white coat, mixing chemicals or looking through a microscope to find a cure for an exotic disease. Well, empirical research about language learning and teaching is similar to that in some ways, but different in many others.
<ul><li>There are many organized and systematic ways of conducting empirical research: </li></ul><ul><li>Questioning </li></ul><ul><li>Eliciting behavior </li></ul><ul><li>Observing/describing </li></ul><ul><li>Experimenting </li></ul><ul><li>This list is certainly not complete. Each form of empirical research offers its own perspective and follows its own set of procedures. These methods will be discussed later in this module. </li></ul>
KINDS OF RESEARCH Generally speaking, in second language research it is useful to distinguish between BASIC (or theoretical), APPLIED , and PRACTICAL research. BASIC RESEARCH is concerned with knowledge for the sake of theory. Its design is not controlled by the practical usefulness of the findings. APPLIED RESEARCH is concerned with showing how the findings can be applied or summarized into some type of teaching methodology. PRACTICAL RESEARCH goes one step further and applies the findings of research to a specific "practical" teaching situation.
A useful way to look at the relationships among these three research types is illustrated in the diagram below. Each of the three different types of research contributes to the other in helping revise and frame the research from each category. For example, practical research may be based on theory that came from previously done basic research. Or, theory may be generated by the combination of results from various practical research projects. The same bidirectional relationship exists between applied research and basic research or practical research.
INTER-RELATIONSHIPS Now that you understand basic, applied, and practical research, let's look at one more way these three types of research work together. Research is like a plant that grows and grows and grows and grows... When it is grown it throws off seeds of all types (basic, applied, and practical), which in turn sprout and create more research projects.
The process continues with all of the new research 'plants‘ throwing off seeds, creating additional, related research projects of various types. Soon there is a body of basic, applied, and practical research projects related to similar topics. And the process goes on and on...
INTRODUCING THE PARAMETERS Because the scope of language research is so broad and there are so many variables involved, it is sometimes difficult to find any hard and fast rules to follow when doing research. On the next few screens you will see a useful set of interrelated and independent PARAMETERS to guide you as you plan language-related research. They are independent in that they can be considered separately. But they are interrelated because in actual practice researchers' choices within one parameter will influence choices in others.
The parameters are... GENERAL APPROACH Synthetic (Holistic) Analytic (Constituent) RESEARCH AIM Deductive (Hypothesis Testing) Heuristic (Hypothesis Generating) CONTROL OVER THE RESEARCH CONTEXT Low High EXPLICITNESS OF DATA COLLECTION PROCEDURES Low High
PARAMETERS IN DEPTH The choices you make in one parameter may reduce your options in the other parameters. In order to understand this system better, let's look at each parameter in a little greater depth. GENERAL APPROACH Synthetic (Holistic) Analytic (Constituent) A synthetic approach to research looks at the research question or topic from a holistic point of view. The researcher tries to understand the parts of the problem by looking at the whole. An analytic approach to research would look at a topic from a constituent point of view. The researcher tries to understand the whole phenomenon by looking at the separate parts.
RESEARCH AIM Deductive (Hypothesis Testing) Heuristic (Hypothesis Generating) The deductive approach is driven by a particular hypothesis. The researcher has a specific, focused statement in mind and his/her objective is to prove or disprove that specific hypothesis. A heuristic approach starts with few preconceived notions or hypotheses about the focus of the research. The researcher observes a phenomenon in order to generate questions or hypotheses for subsequent research.
CONTROL OVER THE RESEARCH CONTEXT Low High A low degree of control would exist in a situation where the researcher does little to affect the context in which the research is carried out. The researcher may observe classes that are already set up. The researcher does not introduce any kind of treatment to the testing group. In a study with high control, the researcher manipulates the research context in various ways. The researcher could choose and arrange the groups to be tested, or a specific treatment could be administered to the subjects.
EXPLICITNESS OF DATA COLLECTION PROCEDURES Low High Many kinds of data are collected by language researchers. Sometimes the data collection procedures or instruments are relatively "loose" or open. Subjects have more latitude in the ways they can respond. Also, there is more room for the personal judgements of the researcher to enter in. Other data-collection procedures or instruments are highly explicit. They follow carefully controlled, objective procedures which allow for little variation in subjects' responses or researchers' interpretations.
KINDS OF RESEARCH Now that you are familiar with the guiding parameters behind research, you are ready to get into the specifics of planning and carrying out your very own research project. Designing research involves deciding on a general approach as well as planning how the various research elements will work together to achieve the desired results.
There are many possible plans and formats for carrying out research. The three major types are: 1. QUALITATIVE 2. DESCRIPTIVE 3. EXPERIMENTAL The "best" type depends on your research question and situation as well as on your basic approach.
QUALITATIVE RESEARCH This type of research goes by many names: ethnography, cognitive anthropology, etc. A good way to understand qualitative research is to examine it in terms of the research parameters we've already discussed: GENERAL APPROACH Synthetic (Holistic) Analytic (Constituent) First, qualitative research tends to be synthetic rather than analytic. It attempts to capture "the big picture" and see how a multitude of variables work together in the real world.
RESEARCH AIM Deductive (Hypothesis Testing) Heuristic (Hypothesis Generating) Another characteristic of qualitative research is that it is generally heuristic or hypothesis generating. Unlike deductive research, it does not start with preconceived notions or hypotheses, attempting to discover, understand, and interpret what is happening in the research context. CONTROL OVER THE RESEARCH CONTEXT Low High In addition, the degree of control over the research context is low. Qualitative research examines naturally occurring behavior, so the investigative methods are as non-intrusive as possible. Therefore, the researcher's effect on the subjects and the data is minimal.
EXPLICITNESS OF DATA COLLECTION PROCEDURES Low High The level of explicitness in data collection procedures is also low. The data are more impressionistic and interpretive than numerical.
DESCRIPTIVE RESEARCH This type of research is also a grouping that includes many particular research methodologies and procedures, such as observations, surveys, self-reports, and tests. The four parameters of research will help us understand how descriptive research in general is similar to, and different from, other types of research.
GENERAL APPROACH Synthetic (Holistic) Analytic (Constituent) Unlike qualitative research, descriptive research may be more analytic. It often focuses on a particular variable or factor. RESEARCH AIM Deductive (Hypothesis Testing) Heuristic (Hypothesis Generating) Descriptive research may also operate on the basis of hypotheses (often generated through previous, qualitative research). That moves it toward the deductive side of the deductive/heuristic continuum.
CONTROL OVER THE RESEARCH CONTEXT Low High Finally, like qualitative research, descriptive research aims to gather data without any manipulation of the research context. In other words, descriptive research is also low on the "control or manipulation of research context" scale. It is non-intrusive and deals with naturally occurring phenomena. EXPLICITNESS OF DATA COLLECTION PROCEDURES Low High In addition, the data collection procedures used in descriptive research may be very explicit. Some observation instruments, for example, employ highly refined categories of behavior and yield quantitative (numerical) data.
These differences also lead to another significant characteristic of descriptive research-the types of subjects it studies. Descriptive research may focus on individual subjects and go into great depth and detail in describing them. Individual variation is not only allowed for but studied. This approach is called a case-study. On the other hand, because of the data collection and analysis procedures (such as surveys) it may employ, descriptive research can also investigate large groups of subjects. Often these are pre-existing classes. In these cases, the analytical procedures tend to produce results that show "average" behavior for the group.
EXPERIMENTAL RESEARCH There are many different types of "experiments." Most are quite different from the common stereotype. All experimental research, however, has several elements in common. One of the most obvious is the division of the subjects into groups (control, experimental, etc.). Another is the use of a "treatment" (usually the independent variable) which is introduced into the research context or manipulated by the researcher. The four research parameters (discussed earlier in this module) will help us understand the other distinguishing characteristics of experimental research.
GENERAL APPROACH Synthetic (Holistic) Analytic (Constituent) On the synthetic-analytic continuum, experimental research tends to fall on the analytic end. Unless it is very complicated, an experiment typically focuses on a specific element (a "constituent part") of the larger process of language learning and teaching. RESEARCH AIM Deductive (Hypothesis Testing) Heuristic (Hypothesis Generating) The next parameter deals with the heuristic (hypothesis-generating) vs. deductive (hypothesis-testing) factor. In contrast to qualitative research, virtually all experiments are designed to test hypotheses.
CONTROL OVER THE RESEARCH CONTEXT Low High Experiments generally fall on the high end of this scale because they attempt to control the research environment to a considerable degree. This can be both a plus and a minus. On the one hand , it allows the researcher to isolate a particular variable and focus on it in order to determine its effect on other variables. Because of this feature, only experimental studies can claim to show any degree of causality. Qualitative and descriptive research can reveal only relationships or processes. On the other hand , control has several disadvantages. One is that it often makes the research situation unnatural. Consequently, subjects may not behave normally in an experiment. Another disadvantage is that it is virtually impossible to control all the variables in a research situation involving human beings. Finally, controlled experiments often raise serious questions about research ethics.
EXPLICITNESS OF DATA COLLECTION PROCEDURES Low High The final parameter deals with the level of explicitness in data collection. Here again, experimental research falls toward the high end of the scale. Carefully focused instruments (tests, observations, questionnaires, etc.) that generate precise quantitative data are the norm in experiments. These data can be analyzed using statistical tests of significance in order to accept or reject the hypothesis.
KINDS OF EXPERIMENTAL RESEARCH Within the realm of experimental research, there are three major types of design: If you choose to conduct experimental research, one of your most important tasks will be to choose the design that gives your research the best combination of internal and external validity. At the same time, it must be practical enough so that you can actually do the research in your own circumstances. TRUE-EXPERIMENTAL QUASI-EXPERIMENTAL PRE-EXPERIMENTAL
Remember, no particular type is right for all situations. Real-world constraints will often dictate what is practical or possible. In any case you need to be careful to recognize the weaknesses of the design you choose. Do not attempt to prove things or make claims in your findings that are beyond the capabilities of your design.
TRUE-EXPERIMENTAL DESIGNS TRUE-EXPERIMENTAL DESIGNS must employ the following: <ul><li>Random selection of subjects </li></ul><ul><li>Use of control groups </li></ul><ul><li>Random assignments to control and experimental groups </li></ul><ul><li>Random assignment of groups to control and experimental conditions </li></ul>
In order for an experiment to follow a true-experimental design, it must meet the preceding criteria. There is some variation in true-experimental designs, but that variation comes in the time(s) that the treatment is given to the experimental group, or in the observation or measurement (pre-test, post-test, mid-test) area. Advantages of the true-experimental design include: <ul><li>Greater internal validity </li></ul><ul><li>Causal claims can be investigated </li></ul>Disadvantages: <ul><li>Less external validity (not like real world conditions) </li></ul><ul><li>Not very practical </li></ul>
QUASI-EXPERIMENTAL DESIGNS QUASI-EXPERIMENTAL DESIGNS are usually constructions that already exist in the real world. Those designs that fall into the quasi-experimental category fall short in some way of the criteria for the true experimental group. A quasi-experimental design will have some sort of control and experimental group, but these groups probably weren't randomly selected. Random selection is usally where true-experimental and quasi-experimental designs differ.
Some advantages of the quasi-experimental design include: <ul><li>Greater external validity (more like real world conditions) </li></ul><ul><li>Much more feasible given time and logistical constraints </li></ul>Disadvantages: <ul><li>Not as many variables controlled (less causal claims) </li></ul>
PRE-EXPERIMENTAL DESIGNS PRE-EXPERIMENTAL DESIGNS are lacking in several areas of the true-experimental criteria. Not only do they lack random selection in most cases, but they usually just employ a single group. This group receives the "treatment," there is no control group. Pilot studies, one-shot case studies, and most research using only one group, fall into this category. The advantages are : <ul><li>Very practical </li></ul><ul><li>Set the stage for further research </li></ul>Disadvantages: <ul><li>Lower validity </li></ul>
RESEARCH QUESTIONS Finding a RESEARCH QUESTION is probably the most important task in the research process because the question becomes the driving force behind the research-from beginning to end. A research question is always stated in question form. It may start out being rather general and become focused and refined later on (after you become more familiar with the topic, learn what others have discovered, define your terms more carefully, etc.)
The research question you start out with forms the basis for your review of related research literature. This general question also evolves into your hypothesis (or focused research question). When you draw conclusions, they should address this question. In the end, the success of your research depends on how well you answer this question.
It is important to choose a question that satisfies certain criteria: <ul><li>It must not be too broad or general (although you will focus it even more later on in the process). </li></ul><ul><li>It shouldn't have already been answered by previous research (although replication with variation is certainly acceptable). </li></ul><ul><li>It ought to be a question that needs to be answered (i.e., the answer will be useful to people). </li></ul><ul><li>It must be a question that can be answered through empirical means. </li></ul>
You can go to many sources to find topics or issues that can lead to research questions. Here are a few: <ul><li>Personal experience </li></ul><ul><li>Professional books </li></ul><ul><li>Articles in professional periodicals </li></ul><ul><li>Professional indexes (LLBA, MLA, ERIC etc.) </li></ul><ul><li>Other teachers and administrators </li></ul><ul><li>Bibliographies of various types </li></ul><ul><li>Unpublished research by others </li></ul>
It is wise to focus your research so that it is "do-able." Be careful! Don't try to do too much in one study. It is, however, very possible (and quite common) to address several related research questions in one study. This approach is "economical" in that it produces more results with about the same amount of effort.
LITERATURE REVIEW LITERATURE REVIEW is a formal survey of professional literature that is pertinent to your particular question. In this way you will find out exactly what others have learned in relation to your question. This process will also help frame and focus your question and move you closer to the hypothesis or focused question. Once you have decide on a general research question, you need to read widely in that area. Use the same sources of information that you consulted when you came up with your general question, but now narrow your focus. Look for information that relates to your research question.
HYPOTHESIS & FOCUSED QUESTION In deductive research, a HYPOTHESIS is necessary. It is focused statement which predicts an answer to your research question. It is based on the findings of previous research (gained from your review of the literature) and perhaps your previous experience with the subject. The ultimate objective of deductive research is to decide whether to accept or reject the hypothesis as stated. When formulating research methods (subjects, data collection instruments, etc.), wise researchers are guided by their hypothesis. In this way, the hypothesis gives direction and focus to the research.
Sometimes researchers choose to state their hypothesis in "null" form. This may seem to run counter to what the researchers really expect, but it is a cautious way to operate. When (and only when) this null hypothesis is disproved or falsified, the researcher may then accept a logically "alternate" hypothesis. This is similar to the procedure used in courts of law. If a person accused of a crime is not shown to be guilty, then it is concluded that he/she is innocent.
In heuristic research , a hypothesis is not necessary. This type of research employs a "discovery approach." In spite of the fact that this type of research does not use a formal hypothesis, focus and structure is still critical. If the research question is too general, the search to find an answer to it may be futile or fruitless. Therefore, after reviewing the relevant literature, the researcher may arrive at a FOCUSED RESEARCH QUESTION .
VARIABLES Very simply, a VARIABLE is a measurable characteristic that varies. It may change from group to group, person to person, or even within one person over time. There are six common variable types: <ul><li>Dependent variables </li></ul><ul><li>Independent variables </li></ul><ul><li>Intervening variables </li></ul><ul><li>Moderator variables </li></ul><ul><li>Control variables </li></ul><ul><li>Extraneous variables </li></ul>
DEPENDENT VARIABLES . . . show the effect of manipulating or introducing the independent variables. For example, if the independent variable is the use or non-use of a new language teaching procedure, then the dependent variable might be students' scores on a test of the content taught using that procedure. In other words, the variation in the dependent variable depends on the variation in the independent variable.
INDEPENDENT VARIABLES . . . are those that the researcher has control over. This "control" may involve manipulating existing variables (e.g., modifying existing methods of instruction) or introducing new variables (e.g., adopting a totally new method for some sections of a class) in the research setting. Whatever the case may be, the researcher expects that the independent variable(s) will have some effect on (or relationship with) the dependent variables.
INTERVENING VARIABLES . . . refer to abstract processes that are not directly observable but that link the independent and dependent variables. In language learning and teaching, they are usually inside the subjects' heads, including various language learning processes which the researcher cannot observe. For example, if the use of a particular teaching technique is the independent variable and mastery of the objectives is the dependent variable, then the language learning processes used by the subjects are the intervening variables.
MODERATOR VARIABLES . . . affect the relationship between the independent and dependent variables by modifying the effect of the intervening variable(s). Unlike extraneous variables, moderator variables are measured and taken into consideration. Typical moderator variables in TESL and language acquisition research (when they are not the major focus of the study) include the sex, age, culture, or language proficiency of the subjects.
CONTROL VARIABLES Language learning and teaching are very complex processes. It is not possible to consider every variable in a single study. Therefore, the variables that are not measured in a particular study must be held constant, neutralized/balanced, or eliminated, so they will not have a biasing effect on the other variables. Variables that have been controlled in this way are called control variables.
EXTRANEOUS VARIABLES . . . are those factors in the research environment which may have an effect on the dependent variable(s) but which are not controlled. Extraneous variables are dangerous. They may damage a study's validity, making it impossible to know whether the effects were caused by the independent and moderator variables or some extraneous factor. If they cannot be controlled, extraneous variables must at least be taken into consideration when interpreting results.
DATA Another very important component is the DATA . Data come in various types. They are a representation of reality, and show the results of measuring properties or processes. Data and the ways they are measured come in various types. One of the most accepted typologies is Stevens' Scales of Measurement. It divides data into four types: NOMINAL , ORDINAL , INTERVAL , RATIO
NOMINAL DATA Nominal means "name bearing." The nominal scale places things into named categories. These things are assigned to groups according to their common or shared elements. For example women who are different in many ways could be assigned to the same category based on their shared gender. Important: The different categories are not ordered in any "more or less" sense. They are just different from each other.
ORDINAL DATA The ordinal scale places things in order. Ordinal data show a particular item's position relative to other items, such as "First, second, third, etc." The ordinal scale doesn't specify the distance between each item. It just puts them in order. For example, in a playground foot race where no one has a watch, the participants will not know their actual times. They will only know who came in first, second, or third.
INTERVAL DATA The interval scale uses equal-sized units of measurement (points, minutes, etc.) and, therefore, shows the distances, or intervals, between subjects' performances. In the foot race example, if the runners' classmates started counting aloud after the first runner crossed the finish line, they might discover that the second place finisher was only two counts behind the winner while the third-place finisher was ten counts behind. Interval data show this difference in distances. Ordinal data would not. It is important to remember that with interval (as opposed to ratio) data, the intervals start from an arbitrary point, not absolute zero.
RATIO DATA Therefore, a student who scores a 60 on a grammar test could not be said to know twice as much grammar as a student who scored 30. Also, the person who scored a 0 on this test would not be said to know no grammar at all. The ratio scale is like the interval scale. It employs equal intervals. However, the ratio scale begins at a true zero point. That point represents an absolute lack of the quality being measured. Because of this characteristic, additional mathematical functions are possible with ratio data that are not possible with other types of data.
Usage note : Strictly speaking, the word data is actually a plural. The singular form (which is rarely used) is datum . Therefore, it is correct to say, for instance, "The data show" or "The data are" rather than "The data shows" or "The data is".
SUBJECTS SUBJECTS are the sources of your data. Most research in language-related fields uses people as subjects. Their characteristics, development, opinions, attitudes, knowledge, performance, etc. are used to answer your research question. In order to choose appropriate subjects you need to decide what your population of interest is.
POPULATIONS In research, population has a specialized meaning. Theoretically, a population is the group from which your subjects are drawn. Therefore, it is also the group that your subjects represent. When discussing your research findings, you must be careful not to generalize your conclusions beyond this group. For this reason, it important to identify the key characteristics of your subjects, and the population they represent. Are they children, teenagers, adults? Are they men or women. What is their nationality? What native language do they speak?
If you already have a sample group with which you will conduct your research, you need to identify their key characteristics carefully in order to produce a corresponding but theoretical population to which you can generalize your findings. On the other hand, if you start with a large population that is too large to work with, then you must select a smaller sample from it. It is extremely important that this sample be representative of the entire group. Ideally, this selection is done through scientific yet random processes.
INSTRUMENTS & PROCEDURES INSTRUMENTS are used to gauge some quality or ability of your subjects. The purpose of the instrument is to elicit the data for your study. In language research, an instrument can be a test, a checklist, a set of categories, etc. The type of instrument and data collection procedure that you use will depend heavily on your choices in the four parameters discussed earlier.
Here are some possible instruments/procedures: <ul><li>Tests of various skills or behaviors (language proficiency in general, particular language skills, psychological traits, etc.) in various formats (multiple-choice, open response, etc.) </li></ul><ul><li>Interviews (unstructured or structured) </li></ul><ul><li>Questionnaires (mailed or in-person) </li></ul><ul><li>Observations of students or teachers </li></ul><ul><li>Diaries kept by language learners or teachers </li></ul><ul><li>Reviews of school records or documents </li></ul><ul><li>Verbal self-reports by learners (introspective or retrospective) </li></ul>
VALIDITY In general, VALIDITY is an indication of how sound your research is. More specifically, validity applies to both the design and the methods of your research. Validity in data collection means that your findings truly represent the phenomenon you are claiming to measure. Valid claims are solid claims. Validity is one of the main concerns with research. "Any research can be affected by different kinds of factors which, while extraneous to the concerns of the research, can invalidate the findings" (Seliger & Shohamy 1989, 95). Controlling all possible factors that threaten the research's validity is a primary responsibility of every good researcher.
INTERNAL VALIDITY is affected by flaws within the study itself such as not controlling some of the major variables (a design problem), or problems with the research instrument (a data collection problem). "Findings can be said to be internally invalid because they may have been affected by factors other than those thought to have caused them, or because the interpretation of the data by the researcher is not clearly supportable" (Seliger & Shohamy 1989, 95).
Here are some factors which affect internal validity: <ul><li>Subject variability </li></ul><ul><li>Size of subject population </li></ul><ul><li>Time given for the data collection or experimental treatment </li></ul><ul><li>History </li></ul><ul><li>Attrition </li></ul><ul><li>Maturation </li></ul><ul><li>Instrument/task sensitivity </li></ul>
EXTERNAL VALIDITY is the extent to which you can generalize your findings to a larger group or other contexts. If your research lacks external validity, the findings cannot be applied to contexts other than the one in which you carried out your research. For example, if the subjects are all males from one ethnic group, your findings might not apply to females or other ethnic groups. Or, if you conducted your research in a highly controlled laboratory envoronment, your findings may not faithfully represent what might happen in the real world. "Findings can be said to be externally invalid because [they] cannot be extended or applied to contexts outside those in which the research took place" (Seliger & Shohamy 1989, 95).
Here are seven important factors affect external validity: <ul><li>Population characteristics (subjects) </li></ul><ul><li>Interaction of subject selection and research </li></ul><ul><li>Descriptive explicitness of the independent variable </li></ul><ul><li>The effect of the research environment </li></ul><ul><li>Researcher or experimenter effects </li></ul><ul><li>Data collection methodology </li></ul><ul><li>The effect of time </li></ul>
ANALYZING DATA Once have your data, you must ANALYZE it. There are many different ways to analyze data: some are simple and some are complex. Some involve grouping, while others involve detailed statistical analysis. The most important thing you do is to choose a method that is in harmony with the parameters you have set and with the kind of data you have collected. Detailed instruction on data analysis is beyond the scope of this module. To learn more about analyzing data, you will need to consult another source: a teacher, a statistician, a good book on the subject, or annother tutorial.
WRITING IT UP Most reports on empirical research follow a standard format. Research on TESL and language acquisition is no exception. This format typically consists of five major sections, which may be subdivided into smaller parts. The five main components of a research report are...
INTRODUCTION <ul><li>The main purpose of the INTRODUCTION is to give a description of the problem that will be addressed. In this section the researcher might discuss the nature of the research, the purpose of the research, the significance of the research problem, and the research question(s) to be addressed. </li></ul><ul><li>RATIONALE </li></ul><ul><li>PURPOSE </li></ul><ul><li>RESEARCH QUESTION(S) </li></ul>Three essential parts of a good introduction are:
<ul><li>RATIONALE Somewhere in the introduction you need to inform the reader of the rationale of your research. This is a brief explanation of why your research topic is worthy of study and may make a significant contribution to the body of already existing research. </li></ul><ul><li>PURPOSE The statement of purpose is not simply a statement of why the research is being done. (That is what the rationale section is for.) Rather, "purpose" refers to the goal or objective of your research. The purpose statement should answer questions. . . </li></ul><ul><li>"What are the objectives of my research?" and </li></ul><ul><li>"What do I expect to discover or learn from this research?" </li></ul>
<ul><li>RESEARCH QUESTION </li></ul><ul><li>The introduction usually ends with a research question or questions. This question should be. . . </li></ul><ul><li>Related to your research purpose </li></ul><ul><li>Focused </li></ul><ul><li>Clear </li></ul>
LITERATURE REVIEW As part of the planning process you should have done a LITERATURE REVIEW , which is a survey of important articles, books and other sources pertaining to your research topic. Now, for the second main section of your research report you need to write a summary of the main studies and research related to your topic. This review of the professional literature relevant to your research question will help to contextualize, or frame, your research. It will also give readers the necessary background to understand your research.
Evaluating other studies: In a review of the literature, you do not merely summarize the research findings that others have reported. You must also evaluate and comment on each study's worth and validity. You may find that some published research is not valid. If it also runs counter to your hypothesis, you may want to critique it in your review. Don't just ignore it. Tell how your research will be better/overcome the flaws. Doing this can strengthen the rationale for conducting your research. Selecting the studies to include in the review: You do not need to report on every published study in the area of your research topic. Choose those studies which are most relevant and most important.
<ul><li>Organizing the review: </li></ul><ul><li>After you have decided which studies to review, you must decide how to order them. In making your selection, keep your research question in mind. It should be your most important guide in determining what other studies are revelant. Many people simple create a list of one-paragraph summaries in chronological order. This is not always the most effective way to organize your review. You should consider other ways, such as... </li></ul><ul><li>By topic </li></ul><ul><li>Problem -> solution </li></ul><ul><li>Cause -> effect </li></ul>
Another approach is to organize your review by argument and counter argument. For example, You may write about those studies that disagree with your hypothesis, and then discuss those that agree with it. Yet another way to organize the studies in your review is to group them according to a particular variable, such as age level of the subjects (child studies, adult studies, etc.) or research method (case studies, experiments, etc.).
The end of the review: The purpose of your review of the literature was to set the stage for your own research. Therefore, you should conclude the review with a statement of your hypothesis, or focused research question. When this is done, you are ready to proceed with part three of your research report, in which you explain the methods you used.
DESIGN & METHOD The DESIGN & METHOD section of the report is where you explain to your reader how you went about carrying out your research. You should describe the subjects, the instruments used, the conditions under which the tests were given, how the tests were scored, how the results were analyzed, etc. Remember that this section needs to be very explicit. A good rule of thumb is to provide enough detail so that others could replicate all the important points of your research. Failure to provide adequate detail may raise doubts in your readers' minds about your procedures and findings.
Make sure you are honest and forthright in this section. For example, if you had some problems with validity, acknowledge the weaknesses in your study so that others can take them into account when they interpret it (and avoid them if they try to replicate it).
RESULTS In the RESULTS of your report you make sense of what you have found. Here you not only present your findings but also talk about the possible reasons for those findings. Also, if your research approach was deductive, then here is where you accept or reject your hypothesis (based on your findings). In addition, in this section you should use your knowledge of the subject in order to make intelligent comments about your results.
BE CAREFUL! Sometimes researchers use this section as a soapbox and talk about things that don't have anything to do with the research that they did. Don't fall into this trap. Make sure your comments are related to (and based on) your research. Do not go beyond your data. Also, as you report and interpret your findings, do not exaggerate or sensationalize them. Nor should you minimize them. A straightforward matter-of-fact style is probably best.
CONCLUSION In the CONCLUSION to your report, you do a number of important things: <ul><li>Summarize the main points you made in your introduction and review of the literature </li></ul><ul><li>Review (very briefly) the research methods and/or design you employed. </li></ul><ul><li>Repeat (in abbreviated form) your findings. </li></ul><ul><li>Discuss the broader implications of those findings. </li></ul><ul><li>Mention the limitations of your research (due to its scope or its weaknesses) </li></ul><ul><li>Offer suggestions for future research related to yours. </li></ul>
ABSTRACT Some research reports end (or begin) with an abstract. An abstract is a highly abbreviated (usually 100-200 words) synopsis of your research. It should describe your rationale and objectives, as well as your methods and findings. Because of its limited length, an abstract cannot go into detail on any of these topics. Nor can it report on the limitations of your research or offer suggestions for future research. For those, readers will have to read the entire report. But, after reading your abstract, people unfamiliar with your research should know what it is about and whether they want to read the entire report.
OTHER RESOURCES PRINT-BASED RESOURCES INTERNET-BASED RESOURCES