Empirical research methods involve collecting empirical observations to answer research questions and test theories. There are common research processes like experiments, case studies, and surveys. Experiments establish causal relationships and confirm theories with a high level of control. Case studies investigate typical cases under realistic conditions with medium control. Surveys investigate information collected from groups with low control. The goal is to summarize data through descriptive statistics and draw inferences about populations through inferential statistics.

1. Empirical Research Methods Poster Check for the latest version at: http://researchmethods.itposter.net
Define research question How facts become theories Common research process Select research method Analyze quantitative data
Confidence Start empirical Statistics is a mathematical science pertaining to the collection, analysis, interpretation or
Can be pretty sure your Research is a systematic explanation, and presentation of data.
in theory is
conclusions are valid ... Relationships increased Empirical research is a research process for answering Experiment Case study Survey
But are you getting at the Why Explanatory
Event between Laws are Prediction is research approach in which questions to solve Statistical methods can be used to summarize or describe a collection of data; this is called
correct issues? research problems and create new
How constructs formed confirmed empirical observations (data) descriptive statistics. In addition, patterns in the data may be modeled in a way that accounts for
Level of control
Purpose
are identified Theory is are collected to answer knowledge. Establishes causal Investigate a typical Investigate information collected randomness and uncertainty in the observations, and then used to draw inferences about the process
How many rejected research question. relationships, confirm »case« in realistic from a group of people, projects, or population being studied; this is called inferential statistics. Both descriptive and inferential
How much Descriptive Research Question (RQ) is what theories. representative conditions. organizations or literature. statistics comprise applied statistics.
The goal of the theory has to you are trying to find out by
When research Event Predictions Research is
undertaking the research
Theory is from theory performed be defined here. Based on the
Who can be research question an empirical Define research process. Control Requires high control Requires medium control Requires low control
formed that
drawn, which Theory is strategy has to be chosen. question Descriptive statistics
Where Exploratory
explains laws
form NOT modified
A clear and precise RQ guides Control on who is using Change to be assessed Technology change is
theory development, research Descriptive statistics are used to describe the basic features of the data in a study. They provide
appropriate
What research hypotheses which technology, (e.g., new technology) implemented across a large
Can be pretty sure you are Event Prediction is It is used to find out what is design, data collection and data when, where, and number of projects. simple summaries about the sample and the measures. Together with simple graphics analysis,
is wide-ranging
When
NOT confirmed already known about a question analysis. they form the basis of virtually every quantitative analysis of data.
getting at the correct Level of access Relationships
Confidence in before trying to answer it.
under which conditions throughout the Description of results,
issues … between Laws are is possible. development process. influence factors,
But can you draw valid constructs formed theory is reduced Review Types of research questions are: Measures of central tendency
Theory is modified To investigate self Assessment in a typical differences and
conclusions? are identified A good literature review is an What?, Where?, Who?, When?, A measure of central tendency is a single number that is used to represent the average score in
important part of any research. Literature How Much?, How many?, Why?, standing tasks from situation required. commonalities is needed.
Event which results can be the distribution.
How? Mode – the most common score in a frequency distribution
obtained immediately.
Research question examples: The goal of literature review is Median – the middlemost score in a distribution
to demonstrate a familiarity Can establish causal Can be incorporated in Can use existing experience. Mean – the common average
What are the key success factors of object-oriented frameworks?
with a body of knowledge and
Does the proposed software improvement increases the efficiency of its users? relationships. normal development Can confirm an effect
How does software development methodology and team size influence developers
productivity?
Create theoretical model establish credibility. Additional,
to show the path to prior
research and how a current
Create
theoretical
Theoretical model is used to
conceptualize the problem
Can confirm theories. activities.
Already scaled up to
generalizes to many
projects/organizations.
Measures of variability
A single number which describes how much the data vary in the distribution.
research is related to it. model stated in research question. It life size if performed on Allow to use standard Range – The difference between the highest and lower score in a distribution.
Research question: »How does software development methodology and team size is commonly represented with real projects. statistical techniques. Variance – The average of the squared deviations from the mean.
causal model. Standard deviation – the square root of the variance, a measure of variability in the
Pro's
Can determine whether Enable research in the
influences developers productivity?« same units as the scores being described.
expected effects apply large.
in studied context.
Consider threats to the research Theoretical model is based on research question and represents set of concepts and
Design research
Other research
methods
Easy to plan.
Help answer why and
Applicable to real world
projects in practice.
Generalization usually
Correlation and regression
Determine associations between two variables.
Correlation – The strength of the relationship between two variables.
relationships between them! Select Practitioner oriented methods how questions. easier.
Threats to the research are related to operationalization and ... Regression – Predicting the value of one variable from another based on the
research Delphi method Can provide qualitative Good for early exploratory correlation.
measurement issues: Action research Insights.
method analysis.
Operationalization issues – The validity of the operationalization Levels Independent Dependent Measures Laboratory oriented methods
Measurement issues – Reliability, validity, sensitivity (see below) (observed variables) Mathematical modeling Application in With little or no May rely on different
variables variables (observed variables) Computer simulation industrial context replication they may projects/organizations Inferential statistics
(latent variables) (latent variables) Laboratory experiment requires give inaccurate results. keeping comparable data.
Technology oriented methods Compromises.
OSSD Proof of technical concept
Difficult to interpret and No control over variables
Inferential statistics or statistical induction comprises the use of statistics to make inferences
Software Design Design case Design generalize (e.g., due to methods.
Literature based methods concerning some unknown aspect of a population.
Lines of code (LOC) experiment study survey confounding factors). Can at most confirm
Con's
RUP development Literature review
H1 Developer per developer per day Statistical analysis association but not causality.
methodology Conceptual study
XP productivity usually not possible. Can be biased due to Sampling distribution – the distribution of means of samples from a population.
Valid but Valid and Few agreed standards differences between Sampling distribution has three important properties:
Reliable but
on procedures for respondents and It has the same mean as the population distribution.
not reliable reliable not valid H2 undertaking case nonrespondents.
Development It has smaller standard deviation as the population distribution.
Measurement relationship Perform research on defined The objective of this activity studies. Questionnaire design may be As the sample size becomes larger, the shape of the distribution approaches a normal
Number of team size – associate latent variables
Reliability threats – refers to the question whether the research can be repeated sample Perform is to run the study according tricky (validity, reliability). distribution, regardless of the shape of the population from which the samples are drawn.
developers with their measures to the study plan.
with the same results. research
collection
General sable population – Process and product Process and product Questionnaires.
Stability reliability – Does the measurement vary over time? Hypothesis testing - is the use of statistics to determine the probability that a given hypothesis
Data
Causal relationships (H1,H2) – define population to which you want to measurement. measurement. Interviews.
Representative reliability – Does the measurement give the same answer when cause-effect relationship between Questionnaires. is true. The usual process of hypothesis testing consists of four steps.
ultimately generalize results. Questionnaires. Project measurement.
applied to all groups? latent variables (theoretical Formulate the null hypothesis H0 (the hypothesis that is of no scientific interest) and the
Equivalence reliability – When there are many measures of the same construct, do All other variables which Interviews Literature survey. alternative hypothesis Ha (statistical term for the research hypothesis).
propositions). Can be tested only by Accessible population –
they all give the same answer? are not the focus of Data is collected with a Identify a test statistic that can be used to assess the truth of the null hypothesis.
evaluating relationships between population that you can actually
research are irrelevant research instrument, for Parametric and Compare case study results Comparing different populations Compute the P-value, which is the probability that a test statistic at least as significant as
observed variables (hypotheses)! gain access.
Analysis
variables. example questionnaire. nonparametric to a representative among respondents, association the one observed would be obtained assuming that the null hypothesis were true. The
types
Validity threats Collect data statistics, compare central comparison baseline: sister and trend analysis, consistency of
Face validity – Research community »good feel«. Actual sample – the sample smaller the P-value, the stronger the evidence against the null hypothesis.
Content validity – Are all aspect of the conceptual variable included in the Variables types actually used in research.
tendencies of treatments,
groups.
project, company baseline,
project subset with no
scores. Compare the p-value to an acceptable significance value alpha (sometimes called an alpha
measurement? value). If p<=alpha, that the observed effect is statistically significant, the null hypothesis is
Hypothesis testing change.
ruled out, and the alternative hypothesis is valid.
Criterion validity – validity is measured against some other standard or measure for Dependent Developer
Hypotheses are tested by comparing LOC
the conceptual variable. efficiency Conclusion validity Internal validity Internal validity
predictions with observed data Represent
Statistical errors
threats
Predictive validity – The measure is known to predict future behavior that is related the »effect« Mean time Analyze data
Major
Observations that confirm a prediction do Internal validity Construct validity Experimental validity or H0 is TRUE H0 is FALSE
to the conceptual variable. Software
not establish the truth of a hypothesis between failure Construct validity External validity reliability
Construct validity – A measure is found to give correct predictions in multiple reliability Wrong decision –
Deductive testing of hypotheses look for Qualitative Construct validity Accept H0 Correct decision
unrelated research processes. This confirm both the theories and the construct validity Quantitative External validity Experimental validity or The null Type II error
disconfirming evidence to falsify Chose data
Independent
of the measure. Requirements {OSSD, RUP, data reliability External validity hypothesis
hypotheses change XP} analysis data Wrong decision –
Conclusion validity – is concerned with the relationship between the treatment and Represent (H0)
Reject H0 Type I error Correct decision
the outcome of the research 8choice of sample size, choice of statistical tests). the »cause«
Experimental validity – (see reliability) Development Number of
Measurement issues team size developers
Depends on data Use Statistical significance – the probability that an experimental result happened by chance.
Sources of invalidity
Internal – Is concerned with the validity within the given environment and the
Reliability - does the measurement give
the same results under the same
conditions (consistency)?
Latent Observed
Use qualitative
data analysis
and the goal of
the study.
quantitative
data analysis
Select statistical test Repeated Here is the Alpha is the probability
reliability of results. It relates to validity of research process design, controls and ANOVA distribution of
Validity - does the measurement method Describe abstract of rejecting the
Logistic Regression
measures. ANOVA values of Z when
Discriminant of
actually provide information about the theoretical concepts. Define ways of measuring Nom Int hypothesis tested when
External – Is the question of how general the findings are. Can you carry over the LoM the hypothesis
conceptual variable? They cannot be latent variables. Level of that hypothesis is true.
Nominal
Inte
research results into actual environment? Level of for tested is true.
Sensitivity - how much does the directly measured. Each latent variable may The objective of this measurement
measurement change with the changes on have multiple empirical measurement for IV 3+ DV? Ord (mean Z = 0)
rval
activity is to analyze How Here we have
Linear
t
Sensitivity
In
the conceptual variable? indicators. the collected data in of DV? many levels Friedman
regression set alpha = 0.05
How much does the measurement change with the change of the conceptual variable? order to answer the does the IV
operationalized study Draw Consider Consider reliability, have
Paired
goal (research validity and sensitivity! The IV is the ? Int -3 -2 -1 0 1 2 3
question). conclusions threats 3+ 2 (related) t-
Values of Zx
N th
variable that test
om er
LoM
O
Consider sources of
START
defines
Qualitative vs. Quantitative analysis
+
for
Positivist research model invalidity (internal,
external)
#
variables
?
2 What
levels of
Int+
conditions
Related
Are
DV? Ord Wilcoxon
Matched
The Z score for an item, indicates how far and in what direction, that item deviates
from its distribution's mean, expressed in units of its distribution's standard deviation.
measurement Int conditions
Qualitative (»Judgments«) Or Pairs
Positivism is a Quantitative (»Hard numbers«) Conclusions can be drawn ? d Power is the probability
independent or
philosophy that World of theory World of propositions Tends to be the poor relation. Easier to implement and collect data. Tick boxes. Or +O Here is the distribution of values of Z when a
statistically or analytically. d+ rd correlated of rejecting the
Problems of opinion and perception when One way
Mind
states that the Operationalization Easier to make comparisons over time and between Disseminate The objective of this activity is to 1 In
t ? particular alternative hypothesis is true hypothesis tested when
Pearson correlation
making the judgment. Int ANOVA
N
places. report the study and its results so
Linear regression
only authentic
Goodness of Fit
results (mean Z = 1)
om
the alternative
Chi-squared
knowledge is The data collected is more likely to create Can be a quick fix when organizations need that external parties are able to Level of Indep. LoM
Nominal
Spearman correlation
+
Beta is the probability of accepting hypothesis is true.
N
scientific differences of opinion over interpretation. performance data to justify project investment. understand the results in their
Linear regression
measurement for
om
Sup the hypothesis tested when the Beta = 0.74
knowledge, and port Not easily measurable. Easier to process through a computer. contexts as well as replicate the ? IV? Kruskal Wallis
/ Fa alternative hypothesis is true. Power = 0.26
lsify
cross tabulation
al
that such Abstract Test As the benefits are longer term, they can be Easier for other stakeholders to examine and study in a different context. How 3+ Ord
rv
Chi-squared
Don't be afraid to talk over
One sample
knowledge can outweighed by shorter term costs. comprehend. many levels
te
ideas with others! does the IV Independent
In
Can lead to inconsistent assessments of
t-test
only come from Real world »Toy world« Trends and patterns easier to identify. -3 -2 -1 0 1 2 3 4
t-test
Reality
positive Laboratory performance between places over time and Can distort the evaluation process as we measure have
Simplify between project elements. ? LoM
Int Values of Zx
affirmation of what is easy to measure. The critical value
Subjective opinions tend to be given less We got an answer to 2 for
theories through Can lead to simplistic judgments and the wider Mann of Z = 1.65
strict scientific status than quantitative ones. more complex picture is ignored. stated research question. IV = Independent Variable IV? Ord
Whitney U
method. End empirical research DV = Dependent Variable
Design Experiment Design case study Design Survey Literature used
Case method facts: Varieties of case study research:
Analyze <Object(s) of study> (what is studied/ Context selection: Does not explicitly control or manipulate variables. Case studies can be carried out by A survey is a study by asking (a group of) people from a population about their opinion on specific Bernd Freimut, Teade Punter, Stefan Biffl, & Marcus Ciolkowski 2002, State-of-the-Art in
observed?) issue with the intention to define relationships outcomes on this issue.
Online vs. Offline Studies a phenomenon in its natural context. taking a positivist or interpretivist Empirical Studies, Virtuelles Software Engineering Kompetenz-zentrum.
for the purpose of <Purpose> (what is the intention?) Student vs. Professional Makes use of qualitative tools and techniques for data approach.
with respect to their <Quality focus> (which effect is Specific vs. general collection and analysis. Can be deductive and inductive. Johnston, R. & Shanks, G. Research Methods in Information Systems. 2003.
studied?) Case study research can be used in a number of different Can use qualitative or quantitative Survey Process: Data analysis Neuman, W. L. 2005, Social research methods : qualitative and quantitative approaches,
from the point of view of the <Perspective> (whose ways. methods. Study definition – determining the goal of a survey. Coding scheme
view?) Can be used for description, discovery and theory testing. Can investigate one or multiple cases. (for open
5th ed. edn.
Logic of sampling Results are generalized Design – operationalizing of the study goals into a set of questions (see theoretical model)
in the context of <Context> (where is the study Implementation – operationalisation of the design so that the survey will be executable.
question) Winston Tellis 1997, "Introduction to Case Study", The Qualitative Report, vol. 3, no. 2.
to population Data entry
conducted?). Population A smaller set of cases a www.wikipedia.org
Execution – the actual data collection and data processing. Checking
researcher selects from a Case research design: In exploratory case studies, fieldwork,
Sampling Analysis – interpretation of the data. Resolve
Sample Sample larger pool and generalizes and data collection may be undertaken
Variables selection: process incomplete data
frame to the population Investigate a phenomenon in depth, get close to the prior to definition of the research Packaging – reporting about the survey results.
Independent and dependent variables
Single
phenomenon, provide a rich description and reveal Statistical testing
case
Observed variables questions and hypotheses.
Measurement scales (nominal, ordinal, interval, ratio)
A list of cases in a
Random sample: a sample in which a researcher
its deep structure. Explanatory cases are suitable for doing
causal studies. In very complex and
multivariate cases, the analysis can make
Types of survey
of results
About the Research Methods Poster
uses a random sampling process so that each Descriptive surveys are frequently conducted to enable descriptive assertions about
Selection of subjects:
Profile description
population or the best sampling element in the population will have an Enable the analysis of data across cases, which use of pattern-matching techniques. some population, i.e., discovering the distribution of certain features or attributes. The This poster is licensed as Creative Commons Attribution-Noncommercial-No Derivative
Multiple
approximation of it. equal probability of being selected. Descriptive cases require that the Question types
enable the researcher to verify that findings are not concern is not about why the observed distribution exists, but instead what that
Works 2.5 Slovenia License
case
Quantity investigator begin with a descriptive Open
the result of idiosyncrasies of the research setting. distribution is.
Separation criteria theory, or face the possibility that Closed
Cross case comparison allows the researcher to use
problems will occur during the project. Explanatory surveys aim at making explanatory claims about the population. For
literal or theoretical replication.
Experiment design: Hypothesis example, when studying how developers use a certain inspection technique, we might want Authors:
Define the set of tests formulation: to explain why some developers prefer one technique while others prefer another. By
How many tests (to make effects visible) Hypoth. statement examining the relationships between different candidate techniques and several Reporting response
Gregor Polančič
Link the design to the hypothesis, measurement scales and statistics H0: positive Case research objectives: explanatory variables, we may try to explain why developers choose one of the techniques. rate Email: info@itposter.net
Randomize, block(a construct that probably has an effect on response) and balance(equal number of subjects) Ha: One or two tailed Total sample
Case study research Explorative surveys are used as a pre-study to a more thorough investigation to assure
Discovery and induction: selected
that important issues are not foreseen. This could be done by creating a loosely structured
Experiment design
Random
Pretest Posttest
Control Experimental Design Discovery is the description and conceptualization of the phenomena. design components:
questionnaire and letting a sample from the population answer to it. The information is
Number located University of Maribor
assignment group group notation Experiment design Conceptualization is achieved by generating hypotheses and developing A study's question. Number
Its propositions, if any. gathered and analyzed, and the results are used to improve the full investigation. In other Faculty of Electrical Engineering and Computer Science
R o x o
Classical o o notation: explanations for observed relationships contacted
Yes Yes Yes Yes Yes words, the explorative survey does not answer the basic research question, but it may
Statements about relationships provide the basis for the building of theory. The unit of analysis.
provide new possibilities that could be analyzed and should therefore be followed up in the
Number returned Institute of Informatics
One shot case study No No Yes No Yes x o X = Treatment The logic linking the Number complete
Testing and deduction: data to propositions. more focused or thorough survey.
One group pretest posttest No Yes Yes No Yes o x o
Static group comparison No No Yes Yes Yes
x o O = Observation Testing is concerned with validating or disconfirming existing theory. The criteria for Poster version: 0.6 (DRAFT)
o Deduction is a means of testing theory according to the natural science interpreting the findings.
Two group posttest only Yes No Yes Yes Yes R
x o R = Random assignment model http://researchmethods.itposter.net
o
Time series design No Yes Yes No Yes o x o x o