The document discusses various methods for evaluating medical information systems and healthcare IT applications. It describes how evaluation aims to assess quality, value, and impacts of IT in healthcare environments. Both formative and summative evaluations are important, with formative providing feedback during development and summative assessing outcomes after implementation. A wide range of quantitative and qualitative methods are presented for different phases of the system development life cycle. The complexity of evaluation in biomedical informatics is also noted.
Observingthedistributionofdiseaseorhealth related events in human population.
• Identify the characteristics with which the disease is associated.
• Basically 3 questions are asked who, when and where.
• Who means the person affected, where means the place and when is the time distribution.
Observingthedistributionofdiseaseorhealth related events in human population.
• Identify the characteristics with which the disease is associated.
• Basically 3 questions are asked who, when and where.
• Who means the person affected, where means the place and when is the time distribution.
TOTAL QUALITY MANAGEMENT (TQM) and it's relevance in healthcare
(A potential question under health care management topic)
CONTENTS:
1. Definitions
2. Milestones of TQM
3. Rationale for quality in healthcare
4. Five attributes of quality
5. Dimensions of quality
6. Quality measurement
7. Quality management and it's principles
8. TQM view in areas of conflict
9. Deming's 14 points
10. PDCA/ Deming's cycle
11. Quality in health care organization
12. Implementation of quality improvement.
13. Quality assurance process
14. Clinical applications of TQM
15. Success story (chokpot PHC)
Happy learning!!
TOTAL QUALITY MANAGEMENT (TQM) and it's relevance in healthcare
(A potential question under health care management topic)
CONTENTS:
1. Definitions
2. Milestones of TQM
3. Rationale for quality in healthcare
4. Five attributes of quality
5. Dimensions of quality
6. Quality measurement
7. Quality management and it's principles
8. TQM view in areas of conflict
9. Deming's 14 points
10. PDCA/ Deming's cycle
11. Quality in health care organization
12. Implementation of quality improvement.
13. Quality assurance process
14. Clinical applications of TQM
15. Success story (chokpot PHC)
Happy learning!!
Deep Random Secrecy is a new cryptologic method designed to preserve the confidentiality of exchanged information even against unlimitedly powered opponents
Running head evaluation tool1evaluation tool6Evaluation Tool.docxcowinhelen
Running head: evaluation tool 1
evaluation tool 6Evaluation Tool
Name
University
Class
Date
Evaluation Tool
Conducting the literature review and the evaluation methodology provided an insight into PICO question (Does implementing a new unified acute and ambulatory EHR (Electronic Health Record) system in the hospital, compared to when they are not used, improve the health care quality for the patients through documentation), and obtaining important information about what needs to be considered in a research project, particularly regarding research tool. The research should consider a tool that proves to be reliable and valid. The researcher should want to know if the tool is accurate and measuring what it is intended to measure (Penfold et al., 2011). Picking the wrong tool for research would result in an incomplete result, hence problem with the evidence. Thus, subsequent researchers may not want to use the flawed methodology to conduct their research. The purpose of this paper is to describe the selected evaluation tool for the project with a rationale, to summarize the criteria used in defining evaluation success, and to develop the assessment plan.Describing the Evaluation Tool Selected for the Evaluation Project
The chosen tool for evaluation is the “Electronic Health Record End User Survey” (AHRQ, n.d.). The tool is a questionnaire that focuses on the usability of an EHR. The questionnaire is designed for the clinical staff in the ambulatory setting to evaluate the usability of an electronic health record in ambulatory care. The aim of the assessment tool is to measure the appropriateness of ambulatory care after the implementation of clinical documentation. The device involves various types of a survey that incorporate many stakeholders who ensure that the hospital adopts new technology relating to the improvement of health care within the hospital. The tool is associated with a survey tool for assessing the EHR implementation based on development initiatives guide. The EHR End User Survey measures the effectiveness realized in the hospital setup through documentation as compared to using the old system of documentation. Based on the developed PICO question that aims at evaluating the benefits that subsume the overtaken documentation. The evaluation tool captures various hospital domains including the end users feedback regarding training and competency, usefulness, usability, infrastructure, and the user support. The tool involves the validation efforts based on needs assessment, the pilot study and the analysis of the nurse respondents. The End User Survey tool based on the EHR provides questionnaire type of review where the clinical staff answer the asked questions focusing on the current state assessment and usability within the hospital. The remote documentation applicable to the new unified ambulatory system makes it easier and efficient since it increases the number o ...
Slide deck from 2008 Symposium "Developing an Expert-System for Health Promotion: An Experimental E-Learning Platform" from the APA-NIOSH International Conference on Work, Stress, and Health
Chapter 5 Program Evaluation and Research TechniquesCharlene R. .docxchristinemaritza
Chapter 5 Program Evaluation and Research Techniques
Charlene R. Weir
Evaluation of health information technology (health IT) programs and projects can range from simple user satisfaction for a new menu or full-scale analysis of usage, cost, compliance, patient outcomes, and observation of usage to data about patient's rate of improvement.
Objectives
At the completion of this chapter the reader will be prepared to:
1.Identify the main components of program evaluation
2.Discuss the differences between formative and summative evaluation
3.Apply the three levels of theory relevant to program evaluation
4.Discriminate program evaluation from program planning and research
5.Synthesize the core components of program evaluation with the unique characteristics of informatics interventions
Key Terms
Evaluation, 72
Formative evaluation, 73
Logic model, 79
Program evaluation, 73
Summative evaluation, 73
Abstract
Evaluation is an essential component in the life cycle of all health IT applications and the key to successful translation of these applications into clinical settings. In planning an evaluation the central questions regarding purpose, scope, and focus of the system must be asked. This chapter focuses on the larger principles of program evaluation with the goal of informing health IT evaluations in clinical settings. The reader is expected to gain sufficient background in health IT evaluation to lead or participate in program evaluation for applications or systems.
Formative evaluation and summative evaluation are discussed. Three levels of theory are presented, including scientific theory, implementation models, and program theory (logic models). Specific scientific theories include social cognitive theories, diffusion of innovation, cognitive engineering theories, and information theory. Four implementation models are reviewed: PRECEDE-PROCEED, PARiHS, RE-AIM, and quality improvement. Program theory models are discussed, with an emphasis on logic models.
A review of methods and tools is presented. Relevant research designs are presented for health IT evaluations, including time series, multiple baseline, and regression discontinuity. Methods of data collection specific to health IT evaluations, including ethnographic observation, interviews, and surveys, are then reviewed.
Introduction
The outcome of evaluation is information that is both useful at the program level and generalizable enough to contribute to the building of science. In the applied sciences, such as informatics, evaluation is critical to the growth of both the specialty and the science. In this chapter program evaluation is defined as the “systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding.”1 Health IT interventions are nearly always embedded in ...
A. Discuss the conditions, advised not conduct an evaluation?
1. There are no questions about the program.
2. The program has no clear direction.
3. Stakeholders cannot agree on the program objectives.
4. There is not enough money to conduct a sound evaluation.
B. Name the stages in a program life cycle?
1. Pilot program stage.
2. Model program stage.
3. Prototype program stage.
4. Organizationally institutionalized program stage.
5. Professionally institutionalized program stage.
C. What do you mean by diversity in the context of health?
Diversity, in the context of health, refers to the numerous ways in which individuals
and groups differ in their beliefs, behaviors, values, backgrounds, preferences, and
biology.
D. What are the steps in conducting the assessment?
1. Involve community members.
2. Define the population.
3. Define the problem to be assessed.
4. Anticipate data- related and methodological issues.
E. List the six planning steps?
1. Team formation and development.
2. Vision creation.
3. Investigation.
4. Prioritization.
5. Decision.
6. Implementation and continuation.
F. What are the types of needs?
1. Expressed need: the problem revealed through healthcare seeking behavior.
2. Normative need: it’s a lack deficit; it’s to know what is the caused.
3. Perceived need: the felt lack as experienced by the target audience.
4. Relative need: identified through a comparison between advantage and disadvantage group.
G. List the Generational Milestones of Evaluation?
1. Technical: growth of scientific management, statistics, and research methodologies.
2. Descriptive: use of goals and objectives as the basis of evaluation to establish program merit
(outcome-focused).
3. Responsiveness: concern for the usefulness of the evaluation to stakeholders participation of
stakeholders in the evaluation process (utilization-focused).
4. Meta-evaluation: evaluation of evaluations across similar programs.
H. List the Standards Established by the Joint Commission on Standards for Educational
Evaluation?
1. Utility: To ensure that the evaluation will meet the content needs of those involved
2. Feasibility: To ensure that the evaluation will be realistic, prudent, diplomatic, and frugal.
3. Propriety: To ensure that the evaluation will be conducted in a legal and unbiased way.
4. Accuracy: To ensure that the evaluation will communicate appropriate and accurate
information.
For more info visit: www.seu-ph-hi.com
SHORT NOTES AND TEST BANK OF PHC 274: Health Planning – BY ALI ALNASSER (MIDTERM EXAM)
- 13 -
I. List the Evaluation Types?
1. Community needs assessment.
2. Process evaluation.
3. Effect evaluation (Outcome evaluation, Impact evaluation).
4. Cost evaluation.
5. Comprehensive evaluation.
6. Meta-evaluation.
J. List The Four Public Health Pyramid?
1. Individual and Direct Services.
2. Community and Enabling Services.
3. Population-Based Services.
4. Infrastructure.
K. List the Public Health Planning Models?
1. PATCH
2. APEX-PH
3. MAPP
4. CHIP
5. PACE-E
M Heenan_PhD Dissertation Lecture_eHealth Lecture_Engaging Leaders in KPI Sel...Mike Heenan
Presentation of the proliferation of measurement in health care and how organizations should redesign indicator selection processes to engage and motivate managers to improve performance. Presentation to eHealth students based on 2023 PhD dissertation.
MEDICAL AUDIT
Evaluation of data, documents, and resources to check performance of systems meets specified standards
PRESCRIPTION MONITORING, ADR, DRUG RELATED PROBLEMS, staff safety, data,defining standards,
collecting data,
identifying areas for improvement,
making necessary changes
back round to defining new standards.
The mission statement sets the direction and priority for developing and implementing the quality plan. It clearly states the nature of the organization’s commitment to quality and should then be tied to the organizational operations through programs, projects, actions and rewards/recognition.
Application Evaluation Project Part 1 Evaluation Plan FocusTec.docxalfredai53p
Application: Evaluation Project Part 1: Evaluation Plan Focus
Technology increases human effectiveness. Using a lever, you can move an object several times your size. In an airplane, you can move exponentially faster than on foot. Using the Internet, you can access information much more quickly than at a library. What possibilities like this exist in the nursing field? What health information technologies can amplify your impact as a nurse far more than ever before? In this Evaluation Project, you will have the opportunity to answer these questions.
Because of the great differences between HIT systems and different goals of an evaluation, there is no one-size-fits-all evaluation plan. Different technologies require different evaluation methods. Consequently, in this part of your Evaluation Project, you will conduct research on how system implementations similar to the one you select have been previously evaluated. After exploring similar system implementations, you will select one research goal and viewpoint to use in the evaluation.
Read the following three scenarios, and select the one that is of most interest to you:
Scenario 1:
Your hospital is implementing a new unified acute and ambulatory Electronic Health Record (EHR) system through which patient care documentation will occur. Interdisciplinary assessment forms (including nursing), clinical decision support, and medical notes will be documented in this system. The implementation of the system is anticipated to improve the hospital’s performance in a multitude of areas. In particular, it is hoped that the use of the EHR system will reduce the rate of patient safety events, improve the quality of care, deter sentinel events, reduce patient readmissions, and impact spending. The implementation of the EHR system is also intended to fulfill the “Meaningful Use” requirements stipulated in the Health Information Technology for Economic and Clinical Health (HITECH) Act. As the hospital’s lead nurse informaticist, you have been tasked with planning the evaluation of the EHR implementation.
Scenario 2:
As the lead nurse informaticist in your hospital, you have been given the task of planning an evaluation for a soon-to-be launched computerized provider order entry (CPOE) system. The CPOE system is designed to replace conventional methods of placing medication, laboratory, admission, referral, and radiology orders. CPOE systems enable health care providers to electronically specify orders, rather than rely on paper prescriptions, telephone calls, and faxes. The intended goal of a CPOE system is to improve safety by ensuring that orders are easily comprehensible through the use of evidence-based order sets. In addition, the CPOE system has the potential for improving workflow by avoiding duplicate orders and reducing the steps between those who place medical orders and their recipients.
Scenario 3:
You are the lead nurse informaticist in a large urban hospital that has recently implemented a new .
Course Objectives:
How does industry use Metrics?
FDA Challenges and Requirements and Use of Quality Metrics
Complexities of Implementation (Industry Feedback)
What does it all mean?
Key in Risk identification – systematic use of information to identify potential sources of harm (hazards)
referring to the risk question or problem description.
Enables the detection of potential problems as early as possible to plan corrective and preventive
actions
Provides indication that controls are losing effectiveness Important in achieving problem resolution and problem prevention
Another important concept of modern quality systems is the use of trending to examine processes as a whole.
Driving wrong behaviors and unintended consequence
• Establishing excessive or overly complex metrics
Comparing data that is not consistently defined or comparison of single data values
Using metrics as a quality “surrogate”
The efficacy of app-supported smartphone interventions for mental health prob...RachitSharma132
to evaluate the efficacy of app-supported smartphone interventions on
a range of mental health outcomes, and to examine whether
various features related to the intervention (theoretical orientation, whether professional guidance was offered, whether
reminders to engage were sent) and sample (degree of mental
health problem) moderated the observed effect sizes
Medical Technology Tackles New Health Care Demand - Research Report - March 2...pchutichetpong
M Capital Group (“MCG”) predicts that with, against, despite, and even without the global pandemic, the medical technology (MedTech) industry shows signs of continuous healthy growth, driven by smaller, faster, and cheaper devices, growing demand for home-based applications, technological innovation, strategic acquisitions, investments, and SPAC listings. MCG predicts that this should reflects itself in annual growth of over 6%, well beyond 2028.
According to Chris Mouchabhani, Managing Partner at M Capital Group, “Despite all economic scenarios that one may consider, beyond overall economic shocks, medical technology should remain one of the most promising and robust sectors over the short to medium term and well beyond 2028.”
There is a movement towards home-based care for the elderly, next generation scanning and MRI devices, wearable technology, artificial intelligence incorporation, and online connectivity. Experts also see a focus on predictive, preventive, personalized, participatory, and precision medicine, with rising levels of integration of home care and technological innovation.
The average cost of treatment has been rising across the board, creating additional financial burdens to governments, healthcare providers and insurance companies. According to MCG, cost-per-inpatient-stay in the United States alone rose on average annually by over 13% between 2014 to 2021, leading MedTech to focus research efforts on optimized medical equipment at lower price points, whilst emphasizing portability and ease of use. Namely, 46% of the 1,008 medical technology companies in the 2021 MedTech Innovator (“MTI”) database are focusing on prevention, wellness, detection, or diagnosis, signaling a clear push for preventive care to also tackle costs.
In addition, there has also been a lasting impact on consumer and medical demand for home care, supported by the pandemic. Lockdowns, closure of care facilities, and healthcare systems subjected to capacity pressure, accelerated demand away from traditional inpatient care. Now, outpatient care solutions are driving industry production, with nearly 70% of recent diagnostics start-up companies producing products in areas such as ambulatory clinics, at-home care, and self-administered diagnostics.
CHAPTER 1 SEMESTER V - ROLE OF PEADIATRIC NURSE.pdfSachin Sharma
Pediatric nurses play a vital role in the health and well-being of children. Their responsibilities are wide-ranging, and their objectives can be categorized into several key areas:
1. Direct Patient Care:
Objective: Provide comprehensive and compassionate care to infants, children, and adolescents in various healthcare settings (hospitals, clinics, etc.).
This includes tasks like:
Monitoring vital signs and physical condition.
Administering medications and treatments.
Performing procedures as directed by doctors.
Assisting with daily living activities (bathing, feeding).
Providing emotional support and pain management.
2. Health Promotion and Education:
Objective: Promote healthy behaviors and educate children, families, and communities about preventive healthcare.
This includes tasks like:
Administering vaccinations.
Providing education on nutrition, hygiene, and development.
Offering breastfeeding and childbirth support.
Counseling families on safety and injury prevention.
3. Collaboration and Advocacy:
Objective: Collaborate effectively with doctors, social workers, therapists, and other healthcare professionals to ensure coordinated care for children.
Objective: Advocate for the rights and best interests of their patients, especially when children cannot speak for themselves.
This includes tasks like:
Communicating effectively with healthcare teams.
Identifying and addressing potential risks to child welfare.
Educating families about their child's condition and treatment options.
4. Professional Development and Research:
Objective: Stay up-to-date on the latest advancements in pediatric healthcare through continuing education and research.
Objective: Contribute to improving the quality of care for children by participating in research initiatives.
This includes tasks like:
Attending workshops and conferences on pediatric nursing.
Participating in clinical trials related to child health.
Implementing evidence-based practices into their daily routines.
By fulfilling these objectives, pediatric nurses play a crucial role in ensuring the optimal health and well-being of children throughout all stages of their development.
QA Paediatric dentistry department, Hospital Melaka 2020Azreen Aj
QA study - To improve the 6th monthly recall rate post-comprehensive dental treatment under general anaesthesia in paediatric dentistry department, Hospital Melaka
Global launch of the Healthy Ageing and Prevention Index 2nd wave – alongside...ILC- UK
The Healthy Ageing and Prevention Index is an online tool created by ILC that ranks countries on six metrics including, life span, health span, work span, income, environmental performance, and happiness. The Index helps us understand how well countries have adapted to longevity and inform decision makers on what must be done to maximise the economic benefits that comes with living well for longer.
Alongside the 77th World Health Assembly in Geneva on 28 May 2024, we launched the second version of our Index, allowing us to track progress and give new insights into what needs to be done to keep populations healthier for longer.
The speakers included:
Professor Orazio Schillaci, Minister of Health, Italy
Dr Hans Groth, Chairman of the Board, World Demographic & Ageing Forum
Professor Ilona Kickbusch, Founder and Chair, Global Health Centre, Geneva Graduate Institute and co-chair, World Health Summit Council
Dr Natasha Azzopardi Muscat, Director, Country Health Policies and Systems Division, World Health Organisation EURO
Dr Marta Lomazzi, Executive Manager, World Federation of Public Health Associations
Dr Shyam Bishen, Head, Centre for Health and Healthcare and Member of the Executive Committee, World Economic Forum
Dr Karin Tegmark Wisell, Director General, Public Health Agency of Sweden
Struggling with intense fears that disrupt your life? At Renew Life Hypnosis, we offer specialized hypnosis to overcome fear. Phobias are exaggerated fears, often stemming from past traumas or learned behaviors. Hypnotherapy addresses these deep-seated fears by accessing the subconscious mind, helping you change your reactions to phobic triggers. Our expert therapists guide you into a state of deep relaxation, allowing you to transform your responses and reduce anxiety. Experience increased confidence and freedom from phobias with our personalized approach. Ready to live a fear-free life? Visit us at Renew Life Hypnosis..
R3 Stem Cells and Kidney Repair A New Horizon in Nephrology.pptxR3 Stem Cell
R3 Stem Cells and Kidney Repair: A New Horizon in Nephrology" explores groundbreaking advancements in the use of R3 stem cells for kidney disease treatment. This insightful piece delves into the potential of these cells to regenerate damaged kidney tissue, offering new hope for patients and reshaping the future of nephrology.
India Clinical Trials Market: Industry Size and Growth Trends [2030] Analyzed...Kumar Satyam
According to TechSci Research report, "India Clinical Trials Market- By Region, Competition, Forecast & Opportunities, 2030F," the India Clinical Trials Market was valued at USD 2.05 billion in 2024 and is projected to grow at a compound annual growth rate (CAGR) of 8.64% through 2030. The market is driven by a variety of factors, making India an attractive destination for pharmaceutical companies and researchers. India's vast and diverse patient population, cost-effective operational environment, and a large pool of skilled medical professionals contribute significantly to the market's growth. Additionally, increasing government support in streamlining regulations and the growing prevalence of lifestyle diseases further propel the clinical trials market.
Growing Prevalence of Lifestyle Diseases
The rising incidence of lifestyle diseases such as diabetes, cardiovascular diseases, and cancer is a major trend driving the clinical trials market in India. These conditions necessitate the development and testing of new treatment methods, creating a robust demand for clinical trials. The increasing burden of these diseases highlights the need for innovative therapies and underscores the importance of India as a key player in global clinical research.
We understand the unique challenges pickleball players face and are committed to helping you stay healthy and active. In this presentation, we’ll explore the three most common pickleball injuries and provide strategies for prevention and treatment.
Leading the Way in Nephrology: Dr. David Greene's Work with Stem Cells for Ki...Dr. David Greene Arizona
As we watch Dr. Greene's continued efforts and research in Arizona, it's clear that stem cell therapy holds a promising key to unlocking new doors in the treatment of kidney disease. With each study and trial, we step closer to a world where kidney disease is no longer a life sentence but a treatable condition, thanks to pioneers like Dr. David Greene.
3. 3
Introduction:
From 1995s in the hope of :
Medical information systems involve:
Computer-
stored databases
• patient
information to
support medical
order entry
• Results
reporting
• Decision support
systems
• clinical
reminders
Comprehensive
systems
• coordinates
patient care
activities by
linking
computer
terminals in
patient care
areas to all
departments
Smaller separate
systems
•Link patient care
areas to only one
department
•Laboratory system
•Radiology system
•Pharmacy system
•Expert systems
•Computerized
databases
CPOE
• Concerns
about patient
safety and
medical error
• Need to Evaluation
Unfortunately reports of system failures have continued
1.Anderson, J.G. and C. Aydin, Evaluating the Organizational Impact of Healthcare Information
Systems. 2005: Springer.pp 5-8.
Hope
• Increasing efficiency, reducing costs, and improving patient care
1995 • Healthcare applications appeared
4. 4
Evaluation Definition:
Evaluation can be defined as “the act of measuring or exploring properties
of a health information system (in planning, development , implementation,
or operation),the result of which informs a decision to be made concerning
that system in a specific context.”
Evaluating should perform not only in technology assessment but also in
the social and behavioral processes
1.Ammenwerth, E., et al., Evaluation of health information systems—problems and challenges.
International Journal of Medical Informatics, 2003. 71(2-3): p. 125-135.
2. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
During IT Evaluation
Not only the technology itself
But also the interaction between IT
and human players in their information
processing
Must be
Considered
5. 5
Evaluation is the means to assess:
5
Quality
Value
Impacts of IT in the health
care environment
Effects
6. 6
Aim of evaluation
:
To provide the basis for a decision about the IT system investigated
Decision-making context is also the context of the evaluation .[2]
1- Yusof, M. M., et al. (2008). "Investigating evaluation frameworks for health information systems." Int J Med Inform
77(6): 377-385.
2-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Press.p 9-11.
Improve performance
Outcomes
Safety
Effectiveness
7. 7
Questions should be answered:
Evaluation seeks to answer the :
Why (objective of evaluation),
Who (which stakeholders’ perspective is going to be evaluated)
When (which phase in the system development life cycle),
What (aspects or focus of evaluation)(What is it going to be used for ?)
How (methods of evaluation) questions [1],[2]
1- Maryati Mohd. Yusof, R.J.P., Lampros K. Stergioulas. Towards a Framework for Health Information Systems
Evaluation. in Proceedings of the 39th Hawaii International Conference on System Sciences. 2006. Hawaii IEEE.
2-SYMONS, V.J., A review of information systems evaluation-content ,context and process. Eur J Inf Syst, 1991. 1(3): p.
205-212.
10. 10
Pre-implementation evaluation
Inform difficult decisions
Prior to starting any program
As well as post-implementation
evaluation
1.Nykanen, P., et al., Guideline for good evaluation practice in health informatics
(GEP-HI). Int J Med Inform, 2011. 80(12): p. 815-27.
11. 11
Assessment:
There are two basic types of assessment:
I. Summative
II. Formative [1]
Measured :
Qualitative and Quantitative:
Putting the results into a metric context .
Qualitatively and subjectively
Quantitatively and objectively :With the aid of a questionnaire
study..[2]
1-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Pr
ess. pp 9-14.
2-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Pre
ss. pp 20-29.
12. 12
Formative and Summativ
e evaluation
Formative evaluation
Throughout the systems
lifecycle
It provides information for
improving the system under
development
Summative evaluation
Focused on assessing the effect or
outcome of the evaluation object
At a certain point of time after
implementation
1.Nykanen, P., et al., Guideline for good evaluation practice in health informatics
(GEP-HI). Int J Med Inform, 2011. 80(12): p. 815-27.
13. 13
Quantitative & qualitative methods:
• Task analysis
• Interface design
• Time motion analysis
• Software log
• Questionnaire
Quantitative methods
• Think aloud protocol
• Unstructured interviews
Qualitative methods
To assess
• user satisfaction
• user-perceived
• usefulness
• usability of this
framework.
Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
14. 14
Categories of most evaluatio
n studies:
I. CPR evaluation studies,
II. Telemedicine evaluation studies,
III. DSS evaluation studies.
Evaluation can be conveniently classified into:
Objectivist and Subjectivist approaches . [2]
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
2- C.P. Friedman, J. Wyatt, Evaluation methods in biomedical informatics, 2nd ed., Springer Science
& Business,Media, New York, US, 2006.
15. 15
Methodology and Methods:
A method is based on a well-defined theory and includes a
consistent set of techniques, tools and principles to
organize it.
Methodology
Method
metric
metric
measure
measure
Method
metric
Method
Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. p 13.
16. 16
Methodology
• Methodology is supposed to:
(1) provide the answer to what to do next, when to do what, and how to do it
(2) to describe the ideas behind such choices and the suppositions (for instance
, the philosophical background) behind them.
A methodology must comprise:
a) The basic philosophies and theories, so that a user of the methodology can
judge the validity of its use
b) Perspective
c) Assumptions
d) Areas of use
e) Applicable methods, tools, and techniques
Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. PP14-15.
17. 17
Complexity of evaluation
in biomedical informatics
1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2006:
Springer .p 6.
Evaluation
methodology
Computer-based
information
systems
Medicine and
healthcare
delivery
Evaluation in
medical
informatics
19. 19
User Requirements Specificati
on
• Analysis of Work Procedures
• Assessment of Bids
• Balanced Scorecard
• BIKVA
• Delphi: Qualitative assessment
• FieM Study
• Focus Group Interview
• Future Workshop
• Grounded Theory
• Heuristic Evaluation
• Interview (no standardized)
• KUBI
• Logical Framework Approach
• Organizational Readiness
• Pardizipp
• Questionnaire (no standardized)
• Requirements Assessment
• Risk Assessment
• Social Network Analysis
• Stakeholder Analysis
• SWOT
• Usability
• Video recording
• WHO: Framework for Assessment of
Strategies
20. 20
Technical Development Phase
Used to provide feed-back for the technical development.
Balanced Score card
Clinical/Diagnostic Performance
Cognitive Assessment
Cognitive Walkthrough
Heuristic Evaluation
Risk Assessment
SWOT
Technical Verification
Think Aloud
Usability
21. 21
Assessment Methods : Adapta
tion Phase
• Analysis of Work Procedures
• BIKVA
• Clinical/Diagnostic Performance
• Cognitive Assessment
• Cognitive Walkthrough
• Equity Implementation Model
• Field Study
• Focus Group Interview
• Functionality Assessment
• Grounded Theory
• Heuristic Evaluation
• Interview (nonstandardized)
• Prospective Time Series
• Questionnaire (nonstandardized)
• RCT, Randomized Controlled Tria
l
• Risk Assessment
• Root Causes Analysis
• Social Network Analysis
• SWOT
• Technical Verification
• Think Aloud
• Usability
• User Acceptance and Satisfaction
• Videorecording
Real operational assessment can take place.
22. 22
Assessment Methods: Evaluat
ion Phase
1. Analysis of Work Procedures
2. Balanced Score card
3. BIKVA
4. Clinical/Diagnostic Performance
5. Cognitive Assessment
6. Cognitive Walkthrough
7. Delphi
8. Equity Implementation Model
9. FieMStudy
10. Focus Group Interview
11. Functionality Assessment
12. Grounded Theory
13. Heuristic Evaluation
14. Impact Assessment
15. Interview (nonstandardized)
16. KUBI
17. Prospective Time Series
18. Questionnaire (nonstandardized)
19. RCT, Randomized Controlled Trial
20. Risk Assessment
21. Root Causes Analysis
22. Social Network Analysis
23. Stakeholder Analysis
24. SWOT
25. Technical Verification
26. Think Aloud
27. Usability
28. User Acceptance and Satisfaction
29. Videorecording
30. WHO: Framework for Assessment
of Strategies
23. 23
• Usability methods, such as heuristic evaluation, cognitive
walk-throughs , RCT and user testing, are increasingly used
to evaluate and improve the design of clinical software
applications.
• Evaluation studies do not focus solely on the structure and
function of information resources; they also address their
impact on persons who are customarily users of these
resources and on the outcomes of users’ interactions with
them to understand users’ actions.
1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2006: Springer .p 6.
2- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic
Press.
24. 24
1- Analysis of Work Procedures
• Assessing what actually happens compared to the expectations.
They may clearly have a role as part of an evaluation study.
• Some well-known options :
1. The Learning Organization
2. Enterprise modeling
3. Business Process Reengineering
4. Use Cases and scenarios
5. Total Quality Management
6. Health Technology Assessment (HTA)
7. Computer-Supported Cooperative Work (CSCW)
8. Cognitive Task Analysis
1- Brender, J., Handbook of Evaluation Methods for Health Informatics
2006: Academic Press.
25. 25
Analysis of Work Procedures
*Note: The use of diagramming techniques and
other forms of graphical modeling requires
experience and understanding the principles
and procedures involved.
Applied in
a health
care setting
Provide
clear
structure:
Framework
• Potential
Views
• Level of
systems
Systems analyses in
a health care
environment
We can use
Diagramming
techniques
26. 26
Enterprise Modeling
Enterprise Architecture (EA) is a strategic activity and
planning tool for an enterprise, which facilitates decision-
making by enabling a conceptual view of the enterprise.
The main objective of an EA approach is to define the
layout of organizational components and relationships
among them.
29. 29
Health Technology Assessme
nt (HTA):
• HTA is concerned with the systematic evaluation of the
consequences of the adoption and use of new health
technologies and improving the evidence on existing
technologies. [1,2]
• One of the basic lessons learned in the area of HCI is
that usability evaluation should start early in the design
process
• Goodman and Ahn (1999) provide a basic overview of
HTA principles and methods. [3]
1. O'Reilly, D., K. Campbell, and R. Goeree, Basics of health technology assessment.
Methods Mol Biol, 2009. 473: p. 263-83.
2. Stevens, A., R. Milne, and A. Burls, Health technology assessment: history and de
mand. J Public Health Med, 2003. 25(2): p. 98-101.
3. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Acade
mic Press.
30. 30
2- Balanced Scorecard
• Created in 1992 by Drs. Robert S. Kaplan and David P.
Norton, the Balanced Scorecard (BSC) is a revolutionary
way to handle strategy management.
• Balancing focus areas by means of a set of indicators for
a set of strategic objectives.
• The Balanced Scorecard is a tool used to measure an
organization’s activities and initiatives against its Vision,
Mission and Values as outlined in its Strategic Plan.
1. http://www.hrh.ca/balancedscorecard, Humber River Hospital, 10-20-2016
2. https://2gc.eu/- Balance Scorecards reports-10-20-2016-
2GC is a strategic execution consultancy with particular experience in
implementing the latest generation Balanced Scorecards. With a global client
list,
31. 31
Balanced Scorecard is a Manage
ment System
Not just a
measurement system
You do not just measure
your heart beat
You use the
heart rate monitor
to manage your exercise
33. 33
3-BIKVA :User Involvement in
Quality
Development A tool for making critical, subjective decisions about an
existing practice.
1. Identifying user satisfaction or dissatisfaction.
2. Summarizing the information in group interview.
3. User encouraged to identify the reasons behind the incidents and
interactions referred to.
4.Interview management in iterative process with the objective of
clarifying all the disparities that relate to the issues of quality
identified by the users.
5.conclusion will then be presented to the decision makers for an
assessment
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic
Press.
34. 34
BIKVA :User Involvement in Q
uality
Development
Developed in Denmark and it is an evaluation and quality
enhancement method .
The evaluation method is interviewing. The evaluation
process starts from the clients, then moves to the front-line
staff (employees in direct contact with the clients) and
finally ends to managers and politicians. Clients are asked
to express and justify "why they are satisfied or dissatisfied
" with the services offered.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
(2016). "Linkki-hanke - yhteys työelämään." Retrieved 11/9/2016, 2016, from :
http://www.palmenia.helsinki.fi/linkki/.
35. 35
3- Questionnaires
Advantage:
• most people can manage it
Different Types :
A. Open questions
B. Checklist questions
C. The Likert scale consisting of a bar with fields to tick on a scale
D. Multipoint scale
E. Semantic differential scale, which in tabular form uses columns wit
h a value scale (for instance, "extremely", "very“,….)
F. Categorical scale
Need verification : need thorough scientific validation of the questi
onnaire in order to provide results leading to optimal action
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic
Press. p 164
36. 36
Questionnaires:
• Questionnaires contain items that are pre-determined by the in
vestigators and consequently are of limited value in identifyi
ng new or emergent issues in the use of a system that the inve
stigators have not previously thought of. [1]
1.Kushniruk, A.W. and V.L. Patel, Cognitive and usability engineering methods
for the evaluation of clinical information systems. J Biomed Inform, 2004. 37(1
): p. 56-76.
37. 37
4- Clinical/Diagnostic Perform
ance Measurement of the diagnostic performance in measures of accuracy
and precision for IT-based expert systems and decision-support
systems before the system is implemented.
Measurement is doing in Technical Development phase but it can
continue during the operational phase (the Adaptation Phase and
Evolution Phase )
The clinical performance of the systems (for diagnostic, prognostic ,
screening tasks, etc.) is typically measured with measures from
medicine, such as accuracy, precision, sensitivity, specificity, and
predictive values.
1.Kaplan, B., Evaluating informatics applications—clinical decision systems. Internationa
l Journal of Medical Informatics, 2001. 64: p. 15-37.
2. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic
Press.
39. 39
*Point*
• Kaplan reviews the literature (only for the years
1997- 1998) on clinical performance of
decision-support systems for the period and
includes other review articles on this subject
and he concluded that most studies use an
experimental or randomized controlled clinical
trials design (RCT) to assess system
performance.
1.Kaplan, B., Evaluating informatics applications—clinical decision systems. International Journal of
Medical Informatics, 2001. 64: p. 15-37.
40. 40
RCT is a trial in which subjects are randomly assigned to two groups:
one (the experimental group) receiving the intervention that is being
tested, and the other (the comparison group or controls) receiving an
alternative treatment.
The two groups are then followed up to see if any differences between
the result. This helps in assessing the effectiveness of the intervention.
40
41. 41
Hierarchy of Study Types
Descriptive
•Case report
•Case series
•Survey
Analytic
Observational
•Cross sectional
•Case-control
•Cohort studies
Experimental
•Randomized
controlled trials
Strength of evidence for causality between a risk factor and outcome
41
42. 42
مطالعات انواع
توصيفي تحليلي
مداخلهاياي مشاهده
باليني ماييزآركا
اجتماعي ماييزآركا
ميداني ماييزآركا
مقطعي
شاهدي مورد
كوهورت
اكولوژيك
موردشرازگ
درمواشرازگ
Goal:
To test
hypotheses
about the
determinants
of disease
In contrast, the goal of intervention
studies is to test the efficacy of
specific treatments or preventive
measures by assigning individual
subjects to one of two or more
treatment or prevention options
42
43. 43
5-RCT, Randomized Controlle
d Trial
Randomized controlled trials are the ideal study design to
evaluate the effectiveness of health-care interventions.
RCT is used to identify marginal differences between two or more
types of treatment.
Useful in assessment studies of IT-based solutions for decision-
support systems and expert systems, only to a limited degree for
other types of systems.
1.Navaneethan, S.D., et al., How to design a randomized controlled trial. Nephrology (Carlton), 2010. 15(8): p.
732-9.
2. Bothwell, L.E. and S.H. Podolsky, The Emergence of the Randomized, Controlled Trial. N Engl J Med, 2016.
375(6): p. 501-4.
P 172
3- 1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
44. 44
All RCTs are controlled clinical trial
BUT
Not all controlled trials are RCTs
Randomization
simple cluster stratified
45. 45
5-RCT, Randomized Controlle
d Trial
An RCT is the most rigorous scientific method for
evaluating the effectiveness of health care
interventions.
However, bias could arise when there are flaws in the
design and management of a trial.
1.Akobeng, A., Understanding randomized controlled trials. Archives of Disease in C
hildhood, 2005. 90(8): p. 840-844.
48. 48
National Cancer Institute at the National Institutes of Health;
https://www.cancer.gov/about-cancer/treatment/clinical-trials/what-are-
trials/randomization; 11/2/2016.
49. 49
RCT usage:
I. By some patients being treated under the old s
ystem and others under the new system, or equ
ally with regard to staff;
II. By some departments keeping the old system a
nd other department getting the new system;
III. Comparing similar departments in different ho
spitals.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
51. 51
RCT IS NOT suitable for:
* ETIOLOGY AND CLINICAL COURSE
smoking and cancer
* RARE & PROLONGED OUTCOME
52. 52
CASP
RCTs - a checklist
• Good randomisation procedures
• Patients blind to treatment
• Clinicians blind to treatment
• All participants followed up
• All participants analysed in the groups to
which they were randomised (intention to
treat)
53. 53
6- Delphi
• Delphi may be characterized as a method for structuring
a group communication process so that the process is
effective in allowing a group of individuals, as a whole,
to deal with a complex problem. [1,3]
• When judgment is necessary with controlled opinion
feedback: we use Delphi
• Qualitative assessment of an effect
• Method for the prediction of the future
• Key advantage : avoids direct confrontation of the
experts.
• Tool for expert problem solving [1,2]
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
2.Okoli, C. and S.D. Pawlowski, The Delphi method as a research tool: an example, design considerations
and applications. Information & Management, 2004. 42(1): p. 15-29.
3-تكنيكدلفي:علوم در آموزش ايراني مجله ،اباذري پروانه ،نصيرياني خديجه ،احمدي اله فضل ،تحقيق در ابزاريبهار ،پزشكيت وابستان-138
7185(:1)8؛
54. 54
Delphi:
• Delphi survey, including guidelines for data collection, d
ata analysis and reporting of results .
• Three ways of Delphi methods:
1- Classic Delphi, 2- Policy Delphi, 3- Decision Delphi
• Preparation of a questionnaire, including the perils and
pitfalls associated with a Questionnaire approach
1.Okoli, C. and S.D. Pawlowski, The Delphi method as a research tool: an example, design considerations a
nd applications. Information & Management, 2004. 42(1): p. 15-29.
2.Kushniruk, A.W. and V.L. Patel, Cognitive and usability engineering methods for the evaluation of clinical informati
on systems. J Biomed Inform, 2004. 37(1): p. 56-76.
3.دلفي تكنيك:تا و بهار ،پزشكي علوم در آموزش ايراني مجله ،اباذري پروانه ،نصيرياني خديجه ،احمدي اله فضل ،تحقيق در ابزاريبستان-1387
185(:1)8؛
59. 59
7-Usability
• Usability evaluation is a method for identifying specific problems with IT p
roducts and specifically focuses on the interaction between the user and tas
k in a defined environment.[1,4]
• The usability evaluation methods are so various which makes the choice of
the appropriate one(s) a difficult task for evaluators. Usability factors are va
rious features that are used to measure how easy systems are in supporting
users’ tasks.[2,3]
1.Brown Iii, W., et al., Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating
mobile health (mHealth) technology. Journal of Biomedical Informatics, 2013. 46(6): p. 1080-1087.
2.Dhouib, A., et al., A classification and comparison of usability evaluation methods for interactive adaptive syst
ems. 2016: p. 246-251.
3.Borycki, E., et al., Usability Methods for Ensuring Health Information Technology Safety: Evidence-Based Ap
proaches Contribution of the IMIA Working Group Health Informatics
for Patient Safety. IMIAYearbook of Medical Informatics, 2013: p. 20-27.
4. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
60. 60
USABILITY EVALUATION METHODS FOR INTER
ACTIVE ADAPTIVE SYSTEMS:
• Cognitive Walkthrough
• Heuristic Evaluation
• Focus Group
• User-as-wizard
• Task experiment based
• Simulated users
1.Dhouib, A., et al., A classification and comparison of usability evaluation meth
ods for interactive adaptive systems. 2016: p. 246-251.
61. 61
Usability Evaluation
Methods
Advantages Disadvantages
Cognitive
walkthrough
(+) Makes predictions
about the design
without
involving users,
(+) Can be used early
in the preliminary evalu
ation phase.
(–)Certain adaptation
aspects not covered,
(–)Time consuming.
Heuristic evaluation (+) Quick and low-cost
evaluation method,
(+) Can be done early
in the IAS development
process
(–) Evaluators must
be experts,
(–) Need to choose
The appropriate
heuristics.
Focus group (+) Fast to conduct,
(+) Can provide large
amounts of data in
short time
(–) Support only
subjective opinions
(–) Depends on
experienced
moderator.
1.Dhouib, A., et al., A classification and comparison of usability evaluation methods for interactive adaptive
systems. 2016: p. 246-251.
62. 62
8- Cognitive Assessment
• Task-oriented
• Assessment of the cognitive aspects of the interaction
between an IT system and its users
• Cognitive aspects deal with the extent to which the system
functions in accordance with the way the users
• Think and work in respect of user interaction with the IT
system in general and with the user dialogue in particular .
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
63. 63
9- Cognitive Walkthrough
• Cognitive Walkthroughs an analytical method designed to evaluate
usability
• Assessment of user 'friendliness' on the basis of a system design,
from specification, to muck-ups and early prototypes of the system,
aimed at judging how well the system complies with the users' way
of thinking
• The cognitive walkthrough is a formalized way of imagining people
’s thoughts and actions when they use an interface for the first time.
1. Wharton, C., et al., The cognitive walkthrough method: a practitioner's guide, in Usability i
nspection methods, N. Jakob and L.M. Robert, Editors. 1994, John Wiley & Sons, Inc.
p. 105-140.
2. A. W. Kushniruk, A. W. Kushniruk, E. M. Borycki-Human, Social, and Organizational Aspe
cts of Health Information Systems-IGI Global, Premier Reference Source (2008)
3. Xiao, T., Broxham, W., Stitzlein, C., Croll, J., & Sanderson, P. Two human-cent
ered approaches to health informatics: Cognitive systems engineering and usa
bility. in Proceedings of the WCC IFIP-IMIA International eHealth Joint Conf
erence. 2010. Brisbane, Qld.
64. 64
Cognitive Walkthrough
• CW is an inspection method for evaluating usability in a user
interface.
• The questions are aids to simulating the user’s cognitive process:
(1)Will the user be trying to achieve the right effect?
(2)Will the user discover that the correct action is available?
(3)Will the user associate the correct action with the desired effect?
(4)If the correct action is performed, will the user see that progress is being
made?
answered with YES or NO and reasons why the user will succeed or fail in
performing the action (“failure/success stories”). [1,2]
1.Blig, L.-O. and A.-L. Osvalder, Enhanced cognitive walkthrough: development of the cogniti
ve walkthrough method to better predict, identify, and present usability problems. Adv. in Hum.-
Comp. Int., 2013. 2013: p. 9-17.
2.Wharton, C., et al., The cognitive walkthrough method: a practitioner's guide, in Usability ins
pection methods, N. Jakob and L.M. Robert, Editors. 1994, John Wiley & Sons, Inc. p. 105
-140.
65. 65
CW
• The method focuses on simplicity in learning, especially
through exploratory learning. CW has been employed to
evaluate medical equipment, such as clinical information
systems, patient information systems, clinical order
systems , dialysis machines, and patient surveillance
systems .[1]
1.Blig, L.-O. and A.-L. Osvalder , Enhanced cognitive walkthrough: development of the c
ognitive walkthrough method to better predict, identify, and present usability problems. A
dv. in Hum.-Comp. Int., 2013. 2013: p. 9-17.
66. 66
10-Focus Group Interview
• A focus group is a form of qualitative research in which
a group of people are asked about their perceptions,
opinions, beliefs, and attitudes towards a product.
• The moderator speaks very little, and encourages the
group to generate the information required by stimulating
discussion through terse provocative statements.
Challenge: analyzing qualitative data
1.Rabiee, F., Focus-group interview and data analysis. Proc Nutr Soc, 2004. 63(4): p. 655-60.
2.Chapter 5: Personal Interviews. Marketing research and information systems. (Marketing and Agribusiness
Texts 2016 [cited 2016 10/31/2016]; Available from: http://www.fao.org/docrep/w3241e/w3241e06.htm.
67. 67
Time of focus group interview
:
During the
early analysis
phases
During
operations
68. 68
The focus used for eliciting descriptive data from population subgroups
Usually, a group of eight to twelve persons are gathered together for a
group interview or discussion on a focused topic.
Focus groups are widely used in the investigation of applied-research
problems and are recognized as a distinct research method.
The method enables researchers to generate new hypotheses;
To explore intermediate variables as a means of explaining certain
relationships found in survey data.
1. Bender, D.E. and D. Ewbank, The focus group as a tool for health research: issues in design and analysis. Health Transit
Rev, 1994. 4(1): p. 63-80.
69. 69
11-Equity Implementation Mo
del (EIM)
• Examine users' reaction to the implementation of a new system
• It may be applied as a measuring instrument to provide the decision-
making basis relation to policy planning and priority setting, as well
as for examination of barriers and drivers at the implementation of
change.
• The EIM helps provide a theory-based understanding for collecting
and reviewing users' reactions to, and acceptance or rejection of, a
new technology or system.
1. Lauer, T.W., K. Joshi, and T. Browdy, Use of the equity implementation model to review clini
cal system implementation efforts: a case report. J Am Med Inform Assoc, 2000. 7(1): p. 91-10
2.
2. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
70. 70
• The method provides a means for explanation of system
implementation events, leading to an understanding of user
resistance or acceptance of the new technology.
• It may be applied as a measuring instrument to provide the decisi
on-making basis relation to policy planning and priority setting, as
well as for examination of barriers and drivers at the implementati
on of change.
71. 71
12- Functionality Assessment
1. Validation of objectives fulfillment
2. Impact Assessment
3. Identification of problems
• It is best suited to qualitative studies
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
72. 72
13-FieMStudy
Observation of an organization to identify its practices
and to expose mechanisms that control change.
This method is widely used in psychology, sociology,
anthropology, and so on that is, in professions that deal
with different perspectives of human factors - to identify
what goes on and how
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
73. 73
14-Grounded Theory
• Grounded Theory is a supportive analytical method for
data acquisition methods that generate textual data, some
open Questionnaire methods, and Interviews.
• This method is subjective.
• In a social context through an iterative and comparative
process in 4 phase:
Open coding
Axial coding
Selective coding
Theoretical coding
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
74. 74
15-Heuristic Evaluation
• Used when no other realizable possibilities exist
• This method can be used for nearly everything but in practice it is m
ost commonly used for the assessment of user interfaces
• Heuristic evaluation, an established evaluation method in the field o
f Human-Computer Interaction, was originally proposed by Jakob N
ielsen .
Five phase:
1. Selection of appropriate heuristics,
2. An individual and independent inspection of the design using heuristics to i
dentify features that conflict with some aspect of best practice,
3. Editing the joint material to identify and resolve duplicates and related findi
ngs,
4. Prioritization to identify severities
5. Analysis and report writing
1.Westbrook, J. and J.I. Westbrook, Information Technology in Health Care 2007: Procee
dings of the 3rd International Conference on Information Technology in Health Care: So
cio-Technical Approaches. 2007: IOS Press . pp 205-207
2- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
76. 76
Zhang and colleagues incorporated Nielsen’s 10 heuristics ,
Shneiderman’s eight golden rules , and the results of their research to
formulate a list of 14 heuristics.
1.Allen, M., et al., Heuristic evaluation of paper-based Web pages: A simplified inspect
ion usability methodology. Journal of Biomedical Informatics, 2006. 39(4): p. 412-423.
77. 77
16- Impact Assessment
• Measurement of the effect plus assessment
of side effect
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
78. 78
17- Interview (Nonstandardized)
• For qualitative studies of subjective
as well as objective circumstances.
1- Brender, J., Handbook of Evaluation Methods for Health
Informatics 2006: Academic Press.
79. 79
18- KUBI
• Translated: Quality development through user involvement
• It has many points in common with the Balanced Scorecard
method
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
80. 80
19- SWOT
• Allows the assessment of an organization from a
neutral perspective through a detailed discussion
of the organization’s strengths, weaknesses, opp
ortunities and threats.
81. 81
SWOT
• The method is useful as a brainstorming method to elucidate a
nd evaluate each of the aspects brought forward.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
82. 82
20- Think Aloud
• Think Aloud is a method that requires users
to speak as they interact with an IT-based
system to solve a problem or perform a
task.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
83. 83
• Thinking aloud technique dates back to the works of experimental
psychology and was first ever described by Karl Duncker (1945) wh
ile he studied productive thinking.
• TAP is popularly used by usability researchers today. But what do
researchers think they get from TAP? Is it right to assume that there
is a one-to-one mapping between verbal protocols and ‘pure data’?
• TAP adds cognitive load on users and can hinder primary tasks
(Preece, 1994). So how do the users experience it? What do users thi
nk of it?
Research Paper: Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. 2002. Getting access to what goes
on in people's heads?: Reflections on the think-aloud technique. In Proceedings of the second Nordic
conference on Human-computer interaction (NordiCHI '02). ACM, New York, NY.
84. 84
• Thinking is much more that what can be explicitly expressed in words.
•To get access to human cognitive processes, a way forward may be to
develop a practice of introspection; to expand our knowledge about the
reflective activity of the user in the expert-guided think aloud session.
•The authors argue that access to subjective experience is possible in
terms of introspection where user has to become a participant in the
analysis of his or her own cognitive processes.
• The paper suggests that use of think aloud should have, as a
prerequisite, explicit descriptions of design, test procedure and
framework for analysis.
85. 85
21- Prospective Time Series
Measurement of a development trend
Time series
Before-and-after studies
Measurement of a number of measures over time shows how an acti
vity or an outcome changes as a function of either time alone or as a
function of different initiatives
This method requires control all over time
Carried out as a matched-pair design in:
Controlled studies including RCTs
Simple before-and-after study
In cohort studies with the follow-up of cases over time
In decision support system with before-after study
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. P 160.
86. 86
22-Risk Assessment
• Monitoring of risk factors in a
development or assessment project to make it
possible to take preemptive action.
• Risk assessment, risk management, and risk
control can be handled either retrospectively
or prospectively.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
87. 87
23-Root Causes Analysis
• Exploration of what, how, and why a given
incident occurred to identify the root causes of
undesirable events.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic
Press.
88. 88
24-Social Network Analysis
• Social network analysis is carried out by means of diagra
mming techniques in relationships between elements wit
hin an organization (such as individuals, professions, de
partments or other organizations)
• Most commonly applied to help improve the effectivenes
s and efficiency of decision making processes in commer
cial organizations.
• The potential value of SNA to measure team function an
d use the information to improve working processes was
another finding of studies in different settings
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
2- Chambers, D., Wilson, P., Thompson, C., & Harden, M. (2012). Social Network Analysis in Health
care Settings: A Systematic Scoping Review. PLoS ONE, 7(8), e41911. http://doi.org/10.1371/journal.
pone.0041911
90. 90
25-Stakeholder Analysis
• Assessment of stakeholder features and their inner dynamics, aimin
g to identify participants for the completion of a given task, a proble
m-solving activity or a project.
Namazzi, G., et al., Stakeholder analysis for a maternal and newborn health project in Eastern Ugand
a. BMC Pregnancy Childbirth, 2013. 13: p. 58.
91. 91
Stakeholder Analysis. Service quality 2016 [cited 2016 11/2/2016]; Available from: http://asq
.org/service/body-of-knowledge/tools-stakeholder-analysis.
92. 92
26-Technical Verification
• Functions are present, work correctly and are in
compliance with the agreement.
A. Completeness of the functionality
B. Correctness of the functionality
C. Coherence of the functionality
D. Consistency of the functionality
E. Interconnectivity of the functionality
F. Reliability
G. Performance
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic
Press.
93. 93
27-User Acceptance and
Satisfaction
Assessment of user opinions, attitudes, and perceptions of an IT syst
em during daily operation
Most common methods :
Interview techniques : Interviews or Focus-Group Interviews.
Questionnaires: most commonly used when measuring user satisfact
ion.
The Equity Implementation Model
*User-satisfaction questionnaires
Aim: verify and correct for aspects such as reliability,
predictive value, accuracy and content validity and
internal validity.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.
94. 94
28-Videorecording
Video recording is a capability for recording videos of
every new Automate test session that is requested on
BrowserStack. While your test is running, you can see a
live screencast that is recorded and saved with a proper
name that can be accessed and downloaded later.
https://www.browserstack.com/question/651
95. 95
29-WHO: Framework for As
sessment of Strategies
The method may be applied for assessment of different (
development) strategies, either individually or as a
comparative analysis.
The method is particularly relevant early in the
explorative Phase for exploring the feasibility of basic
ideas and solutions.
1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.p 222
98. 98
Literature reviews:
• Evaluation studies have been performed with different aims and obje
ctives, and are concerned with different domains and areas of focus.
1.Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in
Healthcare Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 397-
432.
99. 99
Literature reviews about ICT Evaluation
Most important study conducted by Kapalan about CDSS Evaluation[1].
• Conclusion of 27 reviews : RCT is standard method of evaluation approach for
CDSS.
• Problems of RCT: Do not answer questions such as why some systems tend to
be used while others are not
•Unrecognized ineffectiveness reason of CDSS.
1,035 articles reviewed by Ammenwerth and Keizer’s from 1982 to 2002
• Concluded that the number of evaluation studies in the area of medical informatics is
rising significantly.
• appropriateness of patient care, efficiency of patient care, user satisfaction, and software
quality.
• The quality of care processes and patient outcomes was found to have increased.
• Interpreted this shift as a sign of the maturation of evaluation research in medical
informatics.
1. Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in Healthc
are Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 397-432.
100. 10
Delpierre et al reviewed 26 articles about CBPRS , from January
2000 to March 2003.
• Increased satisfaction of users and patients
• Could lead to significant changes in medical practice.
• Most of the studies did not include qualitative factors such as
characteristics of the disease and the tool
Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in He
althcare Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 3
97-432.
101. 10
• Some of the studies in this review evaluate the effects of the
new system’s implementation on the quality of work
performance, such as user job performance , computer
knowledge, and investigation of skill among other users
• Most of the studies included in this paper have used survey
methodology as their research method
• Some of the studies used a clinical trial or cohort study to
research the systems 'outputs.
• No studies have been conducted to explore the impact of IT
on the system as a whole.
104. 10
Free PPT Chart : ALLPPT.com
ALLPPT.com _ Free Powerpoint Templates, Diagrams and Charts
Your own sub headline
Text here
Add text Add
Text here Text here
Add text add text add text add text
add text add text add text add text
Text here Text here
Add text add text add text add text
Text here Text here
Add text add text add text
add text add text add text add text
Text here Text here
Add text add text add text add text
add text add text add text add text
Text here Text here
Add text add text add text add text
Editor's Notes
These applications are referred to generally as medical or clinical information systems or electronic medical records (EMRs).
The word impact is used in the sense of ‘influence’. On the other hand the word ‘effect’ is used in the sense of ‘result’. This is the main difference between the two words impact and effect.
A framework for systems analysis in health care was developed and applied in a health care setting. To provide a clear structure, the framework describes the potential views and levels of systems analyses in a health care environment.
the tool is usually divided into four sections called quadrants. These are designed to help ensure the activities and initiatives being monitored are comprehensive and reflect a well-balanced approach to achieving the Vision
دلفي رويكرد يا روشي سيستماتيك در تحقيق براي استخراج نظرات از يك گروه متخصصان در مورد يك موضوع يا يك سؤال است و يا رسيدن به اجماع گروهي از طريق يك سري از راندهاي پرسشنامه اي با حفظ گمنامي پاسخ دهندگان، و بازخورد نظرات به اعضاي پانل است
The central feature of this method of obtaining information from groups of people is that the interviewer strives to keep the discussion led by a moderator focused upon the issue of concern. The moderator behaves almost like a psycho-therapist who directs the group towards the focus of the researcher. In doing so, the moderator speaks very little, and encourages the group to generate the information required by stimulating discussion through terse provocative statements.
Example of social network: the case of the eye care programme in the Brong Ahafo region, January 2010. Each square represents an actor and the arrow a relationship between two actors (i.e. the existence of a flow of information between two actors). (Source: Karl Blanchet)
By Kapalan Conclusion of 27 reviews : RCT is standard method of evaluation approach for CDSS.
1,035 articles reviewed by Ammenwerth and Keizer’s from 1982 to 2002
Delpierre et al reviewed 26 articles about CBPRS , from January 2000 to March 2003.