· define the terms sample and population and describe some of

· define the terms "sample" and "population" and describe some
of the advantages and disadvantages of using a sample
compared to a population. Support your reasoning with
examples.
· A researcher is studying the effects of caffeine on exam scores
of college students. In this study:
· What would be the population and sample of this research?
Extrapolate your views of the advantages and disadvantages of
samples and populations to this example.
· Why would the researcher want to use a sample or a
population in this study?
· When responding to your classmates' posts, comment on the
examples that they provided to illustrate the advantages and
disadvantages of a sample compared to a population. What
advantages and disadvantages can you add?
Justify your answers with appropriate reasoning and research
from your textbook and course readings.
Health Informatics Research Methods: Principles and Practice,
Second Edition
Chapter 6: Evaluation Methods
© 2017 American Health Information Management Association
1
Learning Objectives
Demonstrate appropriate use of terms related to evaluation
methods and theory.
Explain the rationale for conducting an evaluation or evaluation

research.
Apply evaluation or evaluation research methods to a scenario.
Outline the steps in an evaluation of health information
technology.
2
Introduction: Evaluation and Evaluation Research
Evaluation
A process to determine perceived value by stakeholders
Evaluation research
A specific kind of research that includes social science methods
and a theoretic framework or model as part of exploratory or
hypothesis-driven research
Evaluation Research Methods
Use a theoretic framework
Exploratory or hypothesis driven
Determine the variables based on theoretic constructs
Develop research procedures to study the theoretic constructs
Collect and analyze the data
Forms of Evaluation and Evaluation Research
Two categories of evaluation
Formative evaluation occurs during the program cycle to gather
feedback from stakeholders or end-users to adjust the process or
the software so that the desired effects are achieved.
Summative evaluation occurs at or near the end of the project
and is undertaken to assess effectiveness of the program.

5
Formative Evaluation
Needs assessment
The process of determining, analyzing, and prioritizing needs,
and in turn, identifying and implementing solution strategies to
resolve high-priority needs
Implementation evaluation
Monitoring how well the planned events are actually occurring
Whether the implementation is meeting the expected timeframes
Process evaluation
How well the project was carried out
6
Examples of Formative Evaluation
Improvement in delivery methods with regard to technology
used
The quality of implementation of a new process or technology
Information about the organizational placement of a given
process
How well a planned process aligned with an actual process
7

Summative Evaluation
Two types
Impact evaluation
Assesses the intended or unintended net effects of the program
or technology
Outcome evaluation
Determines whether the program or technology has caused
demonstrable effects as defined in the project goals
8
Summative Evaluation (cont.)
Two components
Cost-effectiveness analysis
Compares the costs to the outcomes of two or more courses of
action
Cost-benefit analysis
Compares the total costs of a program or project with its
benefits
9
Complex Adaptive Systems (CAS)
An entity with many diverse and autonomous components or
parts (called agents) that are interrelated and interdependent
with many interconnections
The agents behave as a unified whole in learning from
experience and in adjusting to changes in the environment.
Each agent maintains itself in an environment which it creates
through its interactions with other agents.

10
Healthcare Organizations as CAS
Healthcare organizations have
Diverse agents that learn such as
Providers, patients, and other stakeholders
Nonlinear interdependencies
Small changes in the system may or may not result in a change
at the system level
Reduce predictability
Self-organization
Patterns emerge based on the interaction of the components
Emergence
Patterns emerge from the system based on the individual agents
as well as the specific interactions undertaken within the system
Coevolution
Both the system and the agent evolve
Sociotechnical Model for Health Information Technology
The eight dimensions provide a means to evaluate aspects of
health information technology.
Each can be assessed for what is working well and what could
be improved.
Evaluation findings can be used to iteratively revise software,
processes, or components of the HIT to improve the use of the
technology or the technology itself to improve safe use of
technology in healthcare.

Design Principles for Health Information Technology
Five rights framework states that optimal design and
implementation will be achieved if we provide:
The right information
To the right person
In the right format
Through the right channe.
At the right time in workflow
The five rights are essentially a framework for formative
evaluation for HIT.
User Centered Design
The product, service, technology, or process should be designed
for the user and their task.
There should be consistency of design elements with other
components of the system.
The language and terminology should be simple and use
terminology that the user is familiar with.
User Centered Design Model
Source: Creately 2016. Reprinted with permission.
User Centered Design (cont.)
The user interface (UI) should be simple and transparent.
The user should know that their actions are successful and
feedback to this effect should be provided.
Information should be provided about how to navigate the
system.

Fundamental aspects of user centered design are to
functionality aligned with the task.
Design the technology to reduce errors.
The System Usability Scale (SUS)
A frequently-used tool that is simple yet informative
Consists of a 10-item questionnaire originally created by John
Brooke in 1986
The SUS can be used to evaluate a wide variety of products and
services including:
Hardware
Software
Mobile devices
Websites
Applications.
17
Evaluation
The evaluator develops the theory of change.
The theory of change is a statement that explains how activities
will produce a series of results and impact and is used to
develop a logic model.
In the logic model the desired effect is identified first and then
the inputs, activities, resources, and outputs required to achieve
these effects are identified.
Logic Model Example

Source: University of Kansas Workgroup for Community Health
and Development 2016. Reprinted with permission.
19
Evaluation Questions
Evaluation questions should be developed to guide the
evaluation.
Questions should be identified and answered based on the logic
models and the critical activities.
By posing the questions, data will be collected to answer these
key questions and important aspects of the project will be
assessed.
Evaluation Data Collection and Activities
Once the evaluation questions are determined, procedures will
need to be established to collect the data to answer the
questions.
Quantitative and qualitative data can be collected through a
variety of methods.
The process of data collection should be feasible and the data
accessible for collection.
Evaluation Research Examples
Process evaluation
O’Malley and colleagues (2005) examined key areas related to
the use of coded data. In this study, the process of inpatient
coding was examined via:

A review of the literature
Flow charting the process
Interviews
Discussion with coders and the users of the coded data
22
Evaluation Research Examples (cont.)
O’Malley and colleagues also provide a useful guide to
calculating accuracy measures for coding. The most common
statistics are:
Sensitivity
Specificity
Positive predictive value
Negative predictive value
The kappa (k) coefficient
23
Evaluation Research Examples (cont.)
Goal-based and impact evaluation
Shah and colleagues (2005) designed a computerized medication
alert system to decrease adverse events with medications but the
alerts often were overridden.
The system was redesigned to improve clinician acceptance
The first goal-based evaluation reflect frequent dismissal
The system was redesigned with user-centered design principals
and re-evaluated for impact
The impact evaluation reflected that high-severity medication
alerts were accepted.

24
Other Evaluation Resources
Office of the National Coordinator (ONC) testing process for
certification of HIT
Agency for Healthcare Research and Quality (AHRQ)
Evaluation Toolkit
Health Informatics Research Methods: Principles and Practice,
Second Edition
Chapter 10: Defining the Research Question and Performing a
Literature Review
Learning Objectives
Formulate research questions for topics in health informatics
and HIM
Articulate clear hypotheses related to research questions
Search knowledge bases such as bibliographic databases
Extract essential information from information sources
Use key terms associated with research questions, hypotheses,
and literature reviews appropriately
Six Components of Systematic Research Process

Defining the research question*
Performing a literature review*
Selecting a research design and method
Collecting data
Analyzing the data
Presenting results
*Components discussed in this chapter; Items 3–6 covered in
subsequent chapters
Importance of Purpose
Knowing the research’s purpose is critical
Purpose
Answer a significant question
Solve a meaningful problem
Contribute to body of knowledge
Drives six components of systematic research process
Development of a Research Question: Using FINER
Feasible Interesting Novel Ethical Relevant
A research question is a clear statement in the form of a
question of the specific issue within a topic that a researcher
wishes to study
Important to write well-developed research questions because
they guide the literature review and the selection of the research
design
Quantitative Process Vs. Qualitative Process
Quantitative process
Generate objective information

Linear
Researchable and specific
Clearly defined at onset of research project is clearly defined
Qualitative process
Interpret or understand phenomena
Iterative or cyclical
Initial preliminary question is refined during research project
Can be ambiguous at onset of research project
6
Sources of Research Questions
Research models
Recommendations of previous researchers
Gaps in the body of knowledge
Problematic areas
Organizations’ problems
Refinement of the Research Question
Refinement: Narrowing to a manageable scope and researchable
question
Feasibility
Additional details learned during literature review
Scope: Breadth of the question
Wide scope is unmanageable
Narrow to an aspect
Researchable question: Observable phenomena that can be
empirically studied
Scale: Extent or pervasiveness

PICO(TS)
Acronym for a well-developed and manageable research
question
Patient, Population, or Problem of interest includes the
condition, subpopulation, and other characteristics and
demographics
Intervention is broadly defined as treatment, program, or other
independent variable manipulated in study
Comparison or Control against which intervention’s effect is
compared
Outcome is specific result of interest, the dependent variable;
should be measurable
Timing is the time frame or duration of interest (optional)
Setting is context, such home or primary care clinic (optional)
Researchers also refer to PICO, PICOS, or PICOT depending
upon the elements they use
PICO(TS) (cont.)
In states with higher levels of managed care penetration, how
does health information exchange, compared to states with
lower levels of managed care, affect Medicaid costs?
Population: States with high levels of managed care
Intervention: Health information exchange
Comparison: States with lower levels of managed care
Outcome: Medicaid costs
Timing and Setting are omitted; however, Timing could be
added as 2010 through 2015 and Setting could be inpatient
hospital

Problem Statement
Quantitative research
Operationalizes
Operational definitions
Qualitative research
Tentative supposition
Guides initial data collection and is revised during the study
based on the data obtained
Single sentence with an action verb that specifically and
succinctly states what the researcher will be doing to investigate
the research question
Limits study’s scope by setting boundaries
Action verb indicates study’s research design
11
Hypothesis
Quantitative
Explicit, testable statement that describes research question in
operational definitions and measurable terms and is written a
priori
Operationalized variables
Prediction of association or difference
Computable measurement
Study’s intention
Qualitative
Tentative or working supposition
Revised during data collection
Iterative and cyclical
Statement of the researchers’ predictions on the study’s
outcome

Based on theory, model, observation, or expectation from
analysis and interpretation of studies in the literature
12
Quantitative Alternative Hypothesis and Null Hypothesis
Direction of Hypothesis
One-tailed hypothesis (test)—predicts direction
Two-tailed hypothesis (test)—no prediction of direction
Choice depends upon
Theory or model
Previous results in literature
Type of study
Purpose Statement
Declarative sentence that summarizes the specific topic and
goals of the research study
Clearly states what the researchers are trying to achieve
Is near beginning of article or research proposal
Explains reason for study, engages the reader, or both
Makes clear how the research will advance the goals of the
reader, audience, or funding agency
Other terms: Aims, objectives, goals, dependent upon journal
and funding agency

15
Literature Review
Literature review is the systematic acquisition, analytical
examination, critical evaluation, and synthesis of important
information about a topic
Comprehensive
Relevant
Literature review has three meanings
1. Process of identifying, reading, summarizing, analyzing, and
synthesizing
2. Dependent product, which is introduction to manuscript or
literature review explaining background of research question
3. Independent product ,which is an expansion of dependent
product into an entire article, book chapter, or book
16
Purposes and Process of Literature Reviews
Purposes
Orient readers and persuade them of necessity of research study
Assure reader that researcher has conducted thorough review of
the topic
Build the researcher’s knowledge of topic resulting in
competencies
Process
Identify sources of information for the literature review
Seek and retrieve the literature
Collect and record information

Analyze and synthesize the information obtained in step 3
© 2017 American Health Information Manageme nt Association
Sources of Information
Printed works
Peer-reviewed journal articles
Books and book chapters
Conference papers
Government documents
Audiovisual media and electronic media
Others
Credibility
Peer review
AACODS (Authority, Accuracy, Coverage, Objectivity, Date,
Significance)
Grey literature
Primary sources and secondary sources
18
Search and Retrieval of Information Sources
Knowledge bases, such as bibliographic databases and digital
collections
Systematic plan to identify
Databases and collections
Search terms
Search multiple and variety of databases
Health informatics and HIM are multidisciplinary
All relevant works not in one bibliographic database

Collection and Recording of Information Sources
Use reference management software
Capture complete citation data at first access to source
Record key information in summary table
Analysis, Evaluation, and Synthesis of Information Sources
Step 4 takes the most time
Summary table can be analytical and evaluative tool by
identifying and emphasizing
Key features
Common characteristics
Trends
Gaps
Chronicling a series of descriptions is inadequate
Synthesizing is required and makes sense of all the information
that has been captured
Comparing similarities and contrasting differences
Critically evaluating studies methods and tools
Interpreting findings
Drawing conclusions
Development of the Literature Review
Characteristics
Comprehensiveness that is relevant and focused
Concise statement of what is known and unknown
Logical and succinct summary from mostly primary sources
Critical analysis and evaluation including strengths,
weaknesses, limitations, and gaps
Synthesis

Conventions
Transparency
Organizational structure
Progression
Equitable coverage
Pertinence
Style
Good literature reviews deliver concentrated knowledge to the
reader
22
Review
Research question: Clear statement, in form of a question, of a
specific issue within the topic the researcher wishes to study
Development of a quantitative research question results in a
researchable and specific question at onset of study;
development of qualitative research question results in initial
preliminary question refined during study
Acronyms FINER and PICO(TS) represent strategies to refine
research questions
Problem statement: Single sentence with action verb that
specifically and succinctly states what will be done to
investigate the research question
Hypothesis: Statement of researchers’ predictions on study’s
outcome. Quantitative versions are null and alternative and can
be one-tailed or two-tailed
Literature review: Systematic acquisition, analytical
examination, critical evaluation, and synthesis of important
information about a topic

Evaluating Research
Excerpts from:
Leedy, P. D., & Ormrod, J. E. (2005). Practical research
planning and design (8th ed.). Upper Saddle
River, NJ: Pearson.
Reflective Questions to Consider When Evaluating Research
1. In what source did you find the researched article?
2. Does the article have a stated research question or problem?
That is, can you determine the
focus of the author’s work?
3. Does the article describe the collection of data, or does is
describe and synthesize other studies
in which data were collected?
4. Is the article logically organized and easy to follow? What
could have been done to improve it’s
organization?

5. Does the article contain a section that outlines and reviews
previous studies on this topic? In
what ways is this previous work relevant to the research
problem?
6. If the author explained procedures that were followed in the
study, are these procedures clear
enough that you could repeat the work and get similar results?
What additional information
might be helpful or essential for you to replicate the study?
7. If data were collected, can you describe how they were
collected and how they were analyzed?
Do you agree with what was done? What additional things
would you have done if you had
been the researcher?
8. Do you agree with the interpretation of the results? Why or
why not?
9. Finally, reflect over the entire article. What is, for you, most
important? What do you find most
interesting? What do you think are the strengths and
weaknesses of this article? Will you
remember this article in the future? Why or why not? (Leedy &
Ormrod, 2005, p.9)
Additional reference for evaluating research in the future:

Pyrczak, F. (2008). Evaluating research in academic jour nals
(4th ed.). Glendale, CA: Pyrczak.
Calculating and Reporting Healthcare Statistics
Chapter 1:
Introduction to Health Statistics
Statistics
What are statistics?
Statistic
A number computed from a larger collection of numbers which
collectively constitute a sample of data
Statistics
A branch of mathematics concerned with collecting, organizing,
summarizing, and analyzing data
Origins of the Term
Originally, “statistics” referred to the collection of information
about and for the “State”
Reasons for Studying Statistics
Decision making

In order to make decisions we must have some information
In healthcare, we use statistics to:
Find out why patients come to the facility
Determine the cost of taking care of patients
Determine the quality of care provided
Meet requirements of accrediting agencies
Comply with requirements of third party payors
Prioritize needed services
Maintain physician specialty mix
Transition from Data to Decision Making
Data → Information → Facts → Improved Understanding →
Better decision making
Data: Unprocessed facts and figures
Data leads to information: Data that are deliberately selected,
processed, and organized to be useful
Information leads to the facts: A piece of information presented
as the truth
Facts lead to improved understanding: What we know
Improved understanding gives individuals the power to make
better decisions
Healthcare Statistics
Health statistics
Provide information about the health of people and their
utilization of healthcare services
Figures used in statistics must be
Relevant
Applicability of the statistics
Reliable

Consistency of results
Descriptive Statistics vs. Inferential Statistics
Descriptive statistics
Describe what the data show about the characteristics of a
sample
Inferential statistics
Help us make inferences, or a guess about a larger group of data
by drawing conclusions from a smaller group of data
The smaller group of data is called a sample
The results obtained from the sample are assumed to be typical
Where Do We Get Health Statistics?
Primary and secondary sources of information
Primary = Health record
Secondary = Disease and operation index (e.g.)
Where Do We Get Health Statistics?
Census
A count of the people and their location
Facility Departments
Keep a count of patients seen and the reason
Vital Statistics (primary information)

Who Uses Health Statistics?
Healthcare administration
Healthcare department managers
Cancer registries
Nursing facilities
Home health agencies
Hospices
Mental health facilities
Who Uses Health Statistics?
Drug and alcohol facilities
Outpatient facilities
Managed care organizations
Healthcare researchers
Accrediting agencies
Government

· define the terms sample and population and describe some of

Recommended

Recommended

More Related Content

Similar to · define the terms sample and population and describe some of

Similar to · define the terms sample and population and describe some of (20)

More from LesleyWhitesidefv

More from LesleyWhitesidefv (20)

Recently uploaded

Recently uploaded (20)

· define the terms sample and population and describe some of