SlideShare a Scribd company logo
1 of 41
Download to read offline
Using Human Factors in Hospital Technology Procurement
Svetlena Taneva, PhD
svetlena@humanfactors.ca
Healthcare Human Factors
University Health Network @healthcarehf
@SvetlenaTaneva
WHAT IS HUMAN FACTORS?
Buxton, W. (2001). Less is More (More or Less), in P. Denning (Ed.). The Invisible Future: The seamless integration of technology in everyday life. New York: McGraw Hill, 145-179.
What is Human Factors?
The application of the science of human
behaviour to the design of systems
Engineering for human efficiency and
effectiveness - accommodate the limits of
the human user.
Bridging the gap between human
capability and demands of the task
210,000 - 400,000 US DEATHS
annually due to preventable medical error
James, J.T. (2013). A new, evidence-based estimate of patient harms associated with hospital care. Journal of Patient Safety, 9(3), 122-128
17,300 – 23,000 CANADIAN DEATHS
annually due to preventable medical error
CIHI (2016). Measuring Patient Harm in Canadian Hospitals. Canadian Patient Safety Institute.
Baker, Norton, et al. (2004). The Canadian Adverse Events Study.
HEALTHCARE TECHNOLOGY PROCUREMENT
The Opportunity
Healthcare environments today:
— Volume of technology increasing
— Complexity of technology increasing
— Interdependency of technologies increasing
— Clinical staff busier
— Increasingly concerned with errors associated with
technology use
Pitfalls of Traditional Procurement
• Cost and functionality
• Stakeholders: lack of front line users
• Vendor demos:
— Very slick
— Biased by how well the vendor is liked
— Biased by how well the vendor organizes the
demo
• Financial decisions are made early on
• Financial review often carries as much weight
as technical and clinical review
HF goals in procurement
— Objective evidence to inform the procurement decision
— Identify risk and needed customizations to inform negotiations
— Inform implementation strategy
— Ensure best fit
Thetestmethodologycallsfor
intendeduserstoperform
representativeandhigh-risktasks.
Usability Testing/Simulations
Elements of a usability test:
— Representative users
— Representative environment (physical constraints)
— Representative tasks with contender products
— Realistic training, realistic use
— Distractions
— Record video and audio
— Qualitative and quantitative
— Semi-structured debrief
Procurement process
CASE STUDY: INFUSION PUMPS
Pump A Pump B
Case Study: Large Volume Infusion Pumps
Stakeholders
— Clinical representation
— Management
— Infection control
— Transportation
— Supply and purchasing
— Biomedical engineering
HF Process
List of
prioritized
tasks
• Stakeholder input via task survey
Shadowing
builds
context
Draft
scenarios
• Clinicians validate and provide example pt. cases
Refine based
on heuristic
evaluation
Pilot and
refine
Usability
Testing
Usability Testing Protocol
— Training
— Setting up and programming the pump
— Modifying the pump settings and hitting a soft limit
— Modifying the pump settings and hitting a hard
limit
— Resolving an occlusion alarm
— Resolving an empty reservoir alarm
— Debrief and questionnaire
Evaluation Criteria
— Task accuracy & error frequency
— Impact and severity of errors
— Task efficiency
— Number of requests for help
— User feedback and preferences
Categorization of Usability Issues
Priority Definition
High
Issues that could have direct patient safety implications and/or
prevent users from completing tasks
Medium
Issues that have a potential indirect patient safety impact and/or
could result in considerable user frustration and inefficiencies
Low
Issues that affect general usability and/or the user’s overall
impression of the system
Number of Usability Issues
Pump A
Pump B
Task Efficiency
Number of Requests
Pump A 18
Pump B 10
Average Time (mm:ss)
Pump A 59:47
Pump B 56:35
Requests for guidance
Time to complete scenarios
0 1 2 3 4
Easiest to use
Best meets my clinical needs
Safest
Number of Participants
Adult Acute Care - Exit Survey
Cardinal Alaris Hospira Symbiq
0 1 2 3 4
Easiest to use
Best meets my clinical needs
Safest
Number of Participants
NICU - Exit Survey
Cardinal Alaris Hospira Symbiq
Questionnaires
Pump A Pump B Pump A Pump B
Overall Comparison Score
Overall Comparison Score
Pump A 10.70
Pump B 18.99
OTHER PROJECTS
HF Informed Procurement Examples
Project
• Electronic medical records
Outcome
Current products on the market are not
easy to use, resulting in use-related
patient safety risks. Decision to
postpone purchase until better
technology is available.
HF value add
Saved close to $1 billion and
prevented introducing
preventable risk.
HF Informed Procurement Examples
Project
• Anesthesia Gas Machines
Outcome
Large variability in performance among
the 4 products evaluated. Two
products stood out as being
significantly easier to use, with
minimal patient safety implications.
Investment made for the
system that is both safe and
best supports the specific
organization's users and
their workflows.
HF value add
HF Informed Procurement Examples
Project
• Surgical Instrument
Management System
Outcome
Systems evaluated performed similarly
across assessment criteria, but one of
them demonstrated a significantly
lesser number of critical patient
safety and usability issues.
Informed the customization
negotiations, training
requirements, mitigations
and implementation
strategy for the chosen
product in order to prevent
any potential use-safety
risk.
HF value add
HF Informed Procurement Examples
Project
• ABG syringes
Outcome
Users were not able to distinguish the
heparinized syringes from non-
heparinized syringes. Packaging,
plunger colour and labeling all
contributed to safety errors, which
perpetuated through supply stocking,
feeds preparation and administration.
The data was used to break
the contract with the
vendor.
HF value add
Benefits: Human Factors Informed Procurement
— Objective evidence to inform the procurement decision
— Identify risks and push for needed customizations
— Improved adoption of technology
— Improved efficiency
— Decreased training
— Improved patient safety
Conclusions
We cannot assume manufacturers test for safety
Simulation takes into consideration the complexity of real healthcare environments
Simulation uncovers system issues that may also be addressed through further simulation
PROCUREMENT LANDSCAPE
Value-Based Procurement
THANK YOU!
How Human Factors Testing Improves Healthcare Technology Procurement

More Related Content

What's hot

Learning from Leadership: how to champion the Hospital Harm Measure and Never...
Learning from Leadership: how to champion the Hospital Harm Measure and Never...Learning from Leadership: how to champion the Hospital Harm Measure and Never...
Learning from Leadership: how to champion the Hospital Harm Measure and Never...Canadian Patient Safety Institute
 
Introduction of the Measuring and Monitoring of Safety (Vincent) Framework to...
Introduction of the Measuring and Monitoring of Safety (Vincent) Framework to...Introduction of the Measuring and Monitoring of Safety (Vincent) Framework to...
Introduction of the Measuring and Monitoring of Safety (Vincent) Framework to...Canadian Patient Safety Institute
 
Patient safety collaborative launch delegate pack
Patient safety collaborative launch delegate packPatient safety collaborative launch delegate pack
Patient safety collaborative launch delegate packNHS Improving Quality
 
Risk Management and Patient Safety Evolution and Progress
Risk Management and Patient Safety Evolution and ProgressRisk Management and Patient Safety Evolution and Progress
Risk Management and Patient Safety Evolution and ProgressPlan de Calidad para el SNS
 
Innovations & Results: What's Working and What Will it Take?
Innovations & Results: What's Working and What Will it Take?Innovations & Results: What's Working and What Will it Take?
Innovations & Results: What's Working and What Will it Take?EngagingPatients
 
Does patient engagement in patient safety and quality committees advance safe...
Does patient engagement in patient safety and quality committees advance safe...Does patient engagement in patient safety and quality committees advance safe...
Does patient engagement in patient safety and quality committees advance safe...Canadian Patient Safety Institute
 
Measuring Patient Harm in Canadian Hospitals and Driving Improvement
Measuring Patient Harm in Canadian Hospitals and Driving ImprovementMeasuring Patient Harm in Canadian Hospitals and Driving Improvement
Measuring Patient Harm in Canadian Hospitals and Driving ImprovementCanadian Patient Safety Institute
 
CU Errors, clinical governance and patient safety
CU Errors, clinical governance and patient safetyCU Errors, clinical governance and patient safety
CU Errors, clinical governance and patient safetyMedic-ELearning
 
Clinical Governance[1]
Clinical Governance[1]Clinical Governance[1]
Clinical Governance[1]Simon Lalonde
 
Patient safety made easy
Patient safety made easyPatient safety made easy
Patient safety made easyNilly Shams
 
Joint Commission and Patients for Patient Safety
Joint Commission and Patients for Patient SafetyJoint Commission and Patients for Patient Safety
Joint Commission and Patients for Patient SafetyPlan de Calidad para el SNS
 
Making it stick: when asking, telling and begging just isn’t enough
Making it stick: when asking, telling and begging just isn’t enoughMaking it stick: when asking, telling and begging just isn’t enough
Making it stick: when asking, telling and begging just isn’t enoughCanadian Patient Safety Institute
 
When being present isn’t enough – Improving patient safety through situationa...
When being present isn’t enough – Improving patient safety through situationa...When being present isn’t enough – Improving patient safety through situationa...
When being present isn’t enough – Improving patient safety through situationa...Canadian Patient Safety Institute
 
Patient Safety Management - ROJoson - 15feb12
Patient Safety Management - ROJoson - 15feb12Patient Safety Management - ROJoson - 15feb12
Patient Safety Management - ROJoson - 15feb12Reynaldo Joson
 
Teesside patient safety conference presentations
Teesside patient safety conference presentationsTeesside patient safety conference presentations
Teesside patient safety conference presentationsNHS Improving Quality
 
Creating a Just Culture of Safety
Creating a Just Culture of SafetyCreating a Just Culture of Safety
Creating a Just Culture of SafetyJane Chiang
 

What's hot (20)

Learning from Leadership: how to champion the Hospital Harm Measure and Never...
Learning from Leadership: how to champion the Hospital Harm Measure and Never...Learning from Leadership: how to champion the Hospital Harm Measure and Never...
Learning from Leadership: how to champion the Hospital Harm Measure and Never...
 
Introduction of the Measuring and Monitoring of Safety (Vincent) Framework to...
Introduction of the Measuring and Monitoring of Safety (Vincent) Framework to...Introduction of the Measuring and Monitoring of Safety (Vincent) Framework to...
Introduction of the Measuring and Monitoring of Safety (Vincent) Framework to...
 
Patient safety collaborative launch delegate pack
Patient safety collaborative launch delegate packPatient safety collaborative launch delegate pack
Patient safety collaborative launch delegate pack
 
Risk Management and Patient Safety Evolution and Progress
Risk Management and Patient Safety Evolution and ProgressRisk Management and Patient Safety Evolution and Progress
Risk Management and Patient Safety Evolution and Progress
 
Innovations & Results: What's Working and What Will it Take?
Innovations & Results: What's Working and What Will it Take?Innovations & Results: What's Working and What Will it Take?
Innovations & Results: What's Working and What Will it Take?
 
Implementing the Vincent Framework at the Frontline
Implementing the Vincent Framework at the FrontlineImplementing the Vincent Framework at the Frontline
Implementing the Vincent Framework at the Frontline
 
Does patient engagement in patient safety and quality committees advance safe...
Does patient engagement in patient safety and quality committees advance safe...Does patient engagement in patient safety and quality committees advance safe...
Does patient engagement in patient safety and quality committees advance safe...
 
Measuring Patient Harm in Canadian Hospitals and Driving Improvement
Measuring Patient Harm in Canadian Hospitals and Driving ImprovementMeasuring Patient Harm in Canadian Hospitals and Driving Improvement
Measuring Patient Harm in Canadian Hospitals and Driving Improvement
 
CU Errors, clinical governance and patient safety
CU Errors, clinical governance and patient safetyCU Errors, clinical governance and patient safety
CU Errors, clinical governance and patient safety
 
Clinical Governance[1]
Clinical Governance[1]Clinical Governance[1]
Clinical Governance[1]
 
Patient safety made easy
Patient safety made easyPatient safety made easy
Patient safety made easy
 
Joint Commission and Patients for Patient Safety
Joint Commission and Patients for Patient SafetyJoint Commission and Patients for Patient Safety
Joint Commission and Patients for Patient Safety
 
Patient safety
Patient safetyPatient safety
Patient safety
 
Making it stick: when asking, telling and begging just isn’t enough
Making it stick: when asking, telling and begging just isn’t enoughMaking it stick: when asking, telling and begging just isn’t enough
Making it stick: when asking, telling and begging just isn’t enough
 
When being present isn’t enough – Improving patient safety through situationa...
When being present isn’t enough – Improving patient safety through situationa...When being present isn’t enough – Improving patient safety through situationa...
When being present isn’t enough – Improving patient safety through situationa...
 
Introduction to patient safety department
Introduction to patient safety departmentIntroduction to patient safety department
Introduction to patient safety department
 
Patient Safety Management - ROJoson - 15feb12
Patient Safety Management - ROJoson - 15feb12Patient Safety Management - ROJoson - 15feb12
Patient Safety Management - ROJoson - 15feb12
 
Teesside patient safety conference presentations
Teesside patient safety conference presentationsTeesside patient safety conference presentations
Teesside patient safety conference presentations
 
Creating a Just Culture of Safety
Creating a Just Culture of SafetyCreating a Just Culture of Safety
Creating a Just Culture of Safety
 
Safe Medication - Think Global, Act Local
Safe Medication - Think Global, Act LocalSafe Medication - Think Global, Act Local
Safe Medication - Think Global, Act Local
 

Similar to How Human Factors Testing Improves Healthcare Technology Procurement

KCR: Cost effectiveness in clinical trials
KCR: Cost effectiveness in clinical trialsKCR: Cost effectiveness in clinical trials
KCR: Cost effectiveness in clinical trialsKCR
 
Usability mHIseminar.Miller-Jacobs
Usability mHIseminar.Miller-JacobsUsability mHIseminar.Miller-Jacobs
Usability mHIseminar.Miller-JacobsmHealth Initiative
 
Computational Modeling & Simulation in Orthopedics: Tools to Comply in an Ev...
Computational Modeling & Simulation in Orthopedics:  Tools to Comply in an Ev...Computational Modeling & Simulation in Orthopedics:  Tools to Comply in an Ev...
Computational Modeling & Simulation in Orthopedics: Tools to Comply in an Ev...April Bright
 
Big data, RWE and AI in Clinical Trials made simple
Big data, RWE and AI in Clinical Trials made simpleBig data, RWE and AI in Clinical Trials made simple
Big data, RWE and AI in Clinical Trials made simpleHadas Jacoby
 
Sociotechnical Aspects: Clinicians and Technology Lecture 2_slides
Sociotechnical Aspects: Clinicians and Technology Lecture 2_slidesSociotechnical Aspects: Clinicians and Technology Lecture 2_slides
Sociotechnical Aspects: Clinicians and Technology Lecture 2_slidesZakCooper1
 
mHealth Israel_Human Factors for MedTech Ergonomics and Usability_Rebecca Moses
mHealth Israel_Human Factors for MedTech Ergonomics and Usability_Rebecca MosesmHealth Israel_Human Factors for MedTech Ergonomics and Usability_Rebecca Moses
mHealth Israel_Human Factors for MedTech Ergonomics and Usability_Rebecca MosesLevi Shapiro
 
Patient Safety And Human Factors Engineering Spring2006
Patient Safety And Human Factors Engineering Spring2006Patient Safety And Human Factors Engineering Spring2006
Patient Safety And Human Factors Engineering Spring2006Carolyn Jenkins
 
Getting Right with The Joint Commission's Communication Goal
Getting Right with The Joint Commission's Communication GoalGetting Right with The Joint Commission's Communication Goal
Getting Right with The Joint Commission's Communication GoalSpok
 
BP Measurement Device Poster
BP Measurement Device PosterBP Measurement Device Poster
BP Measurement Device PosterCaroline Soyars
 
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...MilliporeSigma
 
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...Merck Life Sciences
 
Analytics in healthcare bhuvaneashwar 11th_march
Analytics in healthcare  bhuvaneashwar  11th_marchAnalytics in healthcare  bhuvaneashwar  11th_march
Analytics in healthcare bhuvaneashwar 11th_marchBhuvaneashwar Subramanian
 
ReSuture Pitch Deck Demo Day PDF
ReSuture Pitch Deck Demo Day PDF ReSuture Pitch Deck Demo Day PDF
ReSuture Pitch Deck Demo Day PDF Hannah Eherenfeldt
 
Resume Anthony Donetz 6-2016
Resume Anthony Donetz 6-2016Resume Anthony Donetz 6-2016
Resume Anthony Donetz 6-2016Anthony Donetz
 
CV Resume - Shay Ginsbourg - Regulatory & Testing Affairs Consultant for Gins...
CV Resume - Shay Ginsbourg - Regulatory & Testing Affairs Consultant for Gins...CV Resume - Shay Ginsbourg - Regulatory & Testing Affairs Consultant for Gins...
CV Resume - Shay Ginsbourg - Regulatory & Testing Affairs Consultant for Gins...Shay Ginsbourg
 
How User Experience Design and Human Factors Engineering can Accelerate Produ...
How User Experience Design and Human Factors Engineering can Accelerate Produ...How User Experience Design and Human Factors Engineering can Accelerate Produ...
How User Experience Design and Human Factors Engineering can Accelerate Produ...Greenlight Guru
 

Similar to How Human Factors Testing Improves Healthcare Technology Procurement (20)

KCR: Cost effectiveness in clinical trials
KCR: Cost effectiveness in clinical trialsKCR: Cost effectiveness in clinical trials
KCR: Cost effectiveness in clinical trials
 
Usability mHIseminar.Miller-Jacobs
Usability mHIseminar.Miller-JacobsUsability mHIseminar.Miller-Jacobs
Usability mHIseminar.Miller-Jacobs
 
Computational Modeling & Simulation in Orthopedics: Tools to Comply in an Ev...
Computational Modeling & Simulation in Orthopedics:  Tools to Comply in an Ev...Computational Modeling & Simulation in Orthopedics:  Tools to Comply in an Ev...
Computational Modeling & Simulation in Orthopedics: Tools to Comply in an Ev...
 
Big data, RWE and AI in Clinical Trials made simple
Big data, RWE and AI in Clinical Trials made simpleBig data, RWE and AI in Clinical Trials made simple
Big data, RWE and AI in Clinical Trials made simple
 
Project Cost management in pharmaceuticals
Project Cost management in pharmaceuticalsProject Cost management in pharmaceuticals
Project Cost management in pharmaceuticals
 
Sociotechnical Aspects: Clinicians and Technology Lecture 2_slides
Sociotechnical Aspects: Clinicians and Technology Lecture 2_slidesSociotechnical Aspects: Clinicians and Technology Lecture 2_slides
Sociotechnical Aspects: Clinicians and Technology Lecture 2_slides
 
Savita NYU 2015
Savita NYU 2015Savita NYU 2015
Savita NYU 2015
 
Business model for Microbial Detection
Business model for Microbial DetectionBusiness model for Microbial Detection
Business model for Microbial Detection
 
mHealth Israel_Human Factors for MedTech Ergonomics and Usability_Rebecca Moses
mHealth Israel_Human Factors for MedTech Ergonomics and Usability_Rebecca MosesmHealth Israel_Human Factors for MedTech Ergonomics and Usability_Rebecca Moses
mHealth Israel_Human Factors for MedTech Ergonomics and Usability_Rebecca Moses
 
Patient Safety And Human Factors Engineering Spring2006
Patient Safety And Human Factors Engineering Spring2006Patient Safety And Human Factors Engineering Spring2006
Patient Safety And Human Factors Engineering Spring2006
 
Getting Right with The Joint Commission's Communication Goal
Getting Right with The Joint Commission's Communication GoalGetting Right with The Joint Commission's Communication Goal
Getting Right with The Joint Commission's Communication Goal
 
BP Measurement Device Poster
BP Measurement Device PosterBP Measurement Device Poster
BP Measurement Device Poster
 
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
 
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
Challenges using Multiple Single-use Systems: Functionality versus Extractabl...
 
Analytics in healthcare bhuvaneashwar 11th_march
Analytics in healthcare  bhuvaneashwar  11th_marchAnalytics in healthcare  bhuvaneashwar  11th_march
Analytics in healthcare bhuvaneashwar 11th_march
 
ReSuture Pitch Deck Demo Day PDF
ReSuture Pitch Deck Demo Day PDF ReSuture Pitch Deck Demo Day PDF
ReSuture Pitch Deck Demo Day PDF
 
Resume Anthony Donetz 6-2016
Resume Anthony Donetz 6-2016Resume Anthony Donetz 6-2016
Resume Anthony Donetz 6-2016
 
CV Resume - Shay Ginsbourg - Regulatory & Testing Affairs Consultant for Gins...
CV Resume - Shay Ginsbourg - Regulatory & Testing Affairs Consultant for Gins...CV Resume - Shay Ginsbourg - Regulatory & Testing Affairs Consultant for Gins...
CV Resume - Shay Ginsbourg - Regulatory & Testing Affairs Consultant for Gins...
 
Sappire slides
Sappire slidesSappire slides
Sappire slides
 
How User Experience Design and Human Factors Engineering can Accelerate Produ...
How User Experience Design and Human Factors Engineering can Accelerate Produ...How User Experience Design and Human Factors Engineering can Accelerate Produ...
How User Experience Design and Human Factors Engineering can Accelerate Produ...
 

More from Canadian Patient Safety Institute

Think Global, Act Local: Patient and Family Engagement Strategies & Contribut...
Think Global, Act Local: Patient and Family Engagement Strategies & Contribut...Think Global, Act Local: Patient and Family Engagement Strategies & Contribut...
Think Global, Act Local: Patient and Family Engagement Strategies & Contribut...Canadian Patient Safety Institute
 
Not All Meds Get Along: Reducing Inappropriate Medication Use
Not All Meds Get Along: Reducing Inappropriate Medication Use Not All Meds Get Along: Reducing Inappropriate Medication Use
Not All Meds Get Along: Reducing Inappropriate Medication Use Canadian Patient Safety Institute
 
Acting on Real-Time Patient Reports to Improve Safety: Fraser Health
Acting on Real-Time Patient Reports to Improve Safety: Fraser HealthActing on Real-Time Patient Reports to Improve Safety: Fraser Health
Acting on Real-Time Patient Reports to Improve Safety: Fraser HealthCanadian Patient Safety Institute
 
Acting on Real-Time Patient Reports to Improve Safety: BC Children's
Acting on Real-Time Patient Reports to Improve Safety: BC Children'sActing on Real-Time Patient Reports to Improve Safety: BC Children's
Acting on Real-Time Patient Reports to Improve Safety: BC Children'sCanadian Patient Safety Institute
 
Acting on Real-Time Patient Reports to Improve Safety: Alberta Health Services
Acting on Real-Time Patient Reports to Improve Safety: Alberta Health ServicesActing on Real-Time Patient Reports to Improve Safety: Alberta Health Services
Acting on Real-Time Patient Reports to Improve Safety: Alberta Health ServicesCanadian Patient Safety Institute
 
Webinar 6: Selecting strategies and techniques best suited to address barrier...
Webinar 6: Selecting strategies and techniques best suited to address barrier...Webinar 6: Selecting strategies and techniques best suited to address barrier...
Webinar 6: Selecting strategies and techniques best suited to address barrier...Canadian Patient Safety Institute
 
Webinar 4: Identifying barriers and enablers, and determinants, in theory
Webinar 4: Identifying barriers and enablers, and determinants, in theory 	     Webinar 4: Identifying barriers and enablers, and determinants, in theory
Webinar 4: Identifying barriers and enablers, and determinants, in theory Canadian Patient Safety Institute
 
Webinar 5: Identifying barriers and enablers, and determinants, in practice
Webinar 5: Identifying barriers and enablers, and determinants, in practiceWebinar 5: Identifying barriers and enablers, and determinants, in practice
Webinar 5: Identifying barriers and enablers, and determinants, in practiceCanadian Patient Safety Institute
 
Collaborative “Spaces” and Health Information Technology Design
Collaborative “Spaces” and Health Information Technology Design Collaborative “Spaces” and Health Information Technology Design
Collaborative “Spaces” and Health Information Technology Design Canadian Patient Safety Institute
 
KTIS Webinar 3: Who needs to do what, differently, to promote implementation?
KTIS Webinar 3: Who needs to do what, differently, to promote implementation? KTIS Webinar 3: Who needs to do what, differently, to promote implementation?
KTIS Webinar 3: Who needs to do what, differently, to promote implementation? Canadian Patient Safety Institute
 

More from Canadian Patient Safety Institute (20)

Think Global, Act Local: Patient and Family Engagement Strategies & Contribut...
Think Global, Act Local: Patient and Family Engagement Strategies & Contribut...Think Global, Act Local: Patient and Family Engagement Strategies & Contribut...
Think Global, Act Local: Patient and Family Engagement Strategies & Contribut...
 
Keeping seniors safe
Keeping seniors safeKeeping seniors safe
Keeping seniors safe
 
Indigenous Perspectives on Patient Safety
Indigenous Perspectives on Patient SafetyIndigenous Perspectives on Patient Safety
Indigenous Perspectives on Patient Safety
 
Conquer Silence Webcast - Deck 1 of 2
Conquer Silence Webcast - Deck 1 of 2Conquer Silence Webcast - Deck 1 of 2
Conquer Silence Webcast - Deck 1 of 2
 
Conquer Silence Webcast - Deck 2 of 2
Conquer Silence Webcast - Deck 2 of 2Conquer Silence Webcast - Deck 2 of 2
Conquer Silence Webcast - Deck 2 of 2
 
Récupération optimisée Canada
Récupération optimisée CanadaRécupération optimisée Canada
Récupération optimisée Canada
 
Enhanced Recovery Canada Presentation
Enhanced Recovery Canada PresentationEnhanced Recovery Canada Presentation
Enhanced Recovery Canada Presentation
 
Not All Meds Get Along: Reducing Inappropriate Medication Use
Not All Meds Get Along: Reducing Inappropriate Medication Use Not All Meds Get Along: Reducing Inappropriate Medication Use
Not All Meds Get Along: Reducing Inappropriate Medication Use
 
Acting on Real-Time Patient Reports to Improve Safety: Fraser Health
Acting on Real-Time Patient Reports to Improve Safety: Fraser HealthActing on Real-Time Patient Reports to Improve Safety: Fraser Health
Acting on Real-Time Patient Reports to Improve Safety: Fraser Health
 
Acting on Real-Time Patient Reports to Improve Safety
Acting on Real-Time Patient Reports to Improve SafetyActing on Real-Time Patient Reports to Improve Safety
Acting on Real-Time Patient Reports to Improve Safety
 
Acting on Real-Time Patient Reports to Improve Safety: BC Children's
Acting on Real-Time Patient Reports to Improve Safety: BC Children'sActing on Real-Time Patient Reports to Improve Safety: BC Children's
Acting on Real-Time Patient Reports to Improve Safety: BC Children's
 
Acting on Real-Time Patient Reports to Improve Safety: Alberta Health Services
Acting on Real-Time Patient Reports to Improve Safety: Alberta Health ServicesActing on Real-Time Patient Reports to Improve Safety: Alberta Health Services
Acting on Real-Time Patient Reports to Improve Safety: Alberta Health Services
 
Webinar 6: Selecting strategies and techniques best suited to address barrier...
Webinar 6: Selecting strategies and techniques best suited to address barrier...Webinar 6: Selecting strategies and techniques best suited to address barrier...
Webinar 6: Selecting strategies and techniques best suited to address barrier...
 
Webinar 4: Identifying barriers and enablers, and determinants, in theory
Webinar 4: Identifying barriers and enablers, and determinants, in theory 	     Webinar 4: Identifying barriers and enablers, and determinants, in theory
Webinar 4: Identifying barriers and enablers, and determinants, in theory
 
Webinar 5: Identifying barriers and enablers, and determinants, in practice
Webinar 5: Identifying barriers and enablers, and determinants, in practiceWebinar 5: Identifying barriers and enablers, and determinants, in practice
Webinar 5: Identifying barriers and enablers, and determinants, in practice
 
Collaborative “Spaces” and Health Information Technology Design
Collaborative “Spaces” and Health Information Technology Design Collaborative “Spaces” and Health Information Technology Design
Collaborative “Spaces” and Health Information Technology Design
 
WHO Clean Hands "It's in your hands"
WHO Clean Hands "It's in your hands"WHO Clean Hands "It's in your hands"
WHO Clean Hands "It's in your hands"
 
Complexities of hand hygiene by GOJO
Complexities of hand hygiene by GOJOComplexities of hand hygiene by GOJO
Complexities of hand hygiene by GOJO
 
KTIS Webinar 3: Who needs to do what, differently, to promote implementation?
KTIS Webinar 3: Who needs to do what, differently, to promote implementation? KTIS Webinar 3: Who needs to do what, differently, to promote implementation?
KTIS Webinar 3: Who needs to do what, differently, to promote implementation?
 
KTIS Webinar 2: Knowledge creation & synthesis
KTIS Webinar 2: Knowledge creation & synthesis KTIS Webinar 2: Knowledge creation & synthesis
KTIS Webinar 2: Knowledge creation & synthesis
 

Recently uploaded

Bangalore Call Girls Majestic 📞 9907093804 High Profile Service 100% Safe
Bangalore Call Girls Majestic 📞 9907093804 High Profile Service 100% SafeBangalore Call Girls Majestic 📞 9907093804 High Profile Service 100% Safe
Bangalore Call Girls Majestic 📞 9907093804 High Profile Service 100% Safenarwatsonia7
 
Call Girls In Andheri East Call 9920874524 Book Hot And Sexy Girls
Call Girls In Andheri East Call 9920874524 Book Hot And Sexy GirlsCall Girls In Andheri East Call 9920874524 Book Hot And Sexy Girls
Call Girls In Andheri East Call 9920874524 Book Hot And Sexy Girlsnehamumbai
 
VIP Call Girls Pune Vani 9907093804 Short 1500 Night 6000 Best call girls Ser...
VIP Call Girls Pune Vani 9907093804 Short 1500 Night 6000 Best call girls Ser...VIP Call Girls Pune Vani 9907093804 Short 1500 Night 6000 Best call girls Ser...
VIP Call Girls Pune Vani 9907093804 Short 1500 Night 6000 Best call girls Ser...Miss joya
 
Russian Call Girls in Bangalore Manisha 7001305949 Independent Escort Service...
Russian Call Girls in Bangalore Manisha 7001305949 Independent Escort Service...Russian Call Girls in Bangalore Manisha 7001305949 Independent Escort Service...
Russian Call Girls in Bangalore Manisha 7001305949 Independent Escort Service...narwatsonia7
 
Call Girl Chennai Indira 9907093804 Independent Call Girls Service Chennai
Call Girl Chennai Indira 9907093804 Independent Call Girls Service ChennaiCall Girl Chennai Indira 9907093804 Independent Call Girls Service Chennai
Call Girl Chennai Indira 9907093804 Independent Call Girls Service ChennaiNehru place Escorts
 
Call Girls Chennai Megha 9907093804 Independent Call Girls Service Chennai
Call Girls Chennai Megha 9907093804 Independent Call Girls Service ChennaiCall Girls Chennai Megha 9907093804 Independent Call Girls Service Chennai
Call Girls Chennai Megha 9907093804 Independent Call Girls Service ChennaiNehru place Escorts
 
Call Girl Coimbatore Prisha☎️ 8250192130 Independent Escort Service Coimbatore
Call Girl Coimbatore Prisha☎️  8250192130 Independent Escort Service CoimbatoreCall Girl Coimbatore Prisha☎️  8250192130 Independent Escort Service Coimbatore
Call Girl Coimbatore Prisha☎️ 8250192130 Independent Escort Service Coimbatorenarwatsonia7
 
Call Girls Service In Shyam Nagar Whatsapp 8445551418 Independent Escort Service
Call Girls Service In Shyam Nagar Whatsapp 8445551418 Independent Escort ServiceCall Girls Service In Shyam Nagar Whatsapp 8445551418 Independent Escort Service
Call Girls Service In Shyam Nagar Whatsapp 8445551418 Independent Escort Serviceparulsinha
 
Russian Call Girls Chennai Madhuri 9907093804 Independent Call Girls Service ...
Russian Call Girls Chennai Madhuri 9907093804 Independent Call Girls Service ...Russian Call Girls Chennai Madhuri 9907093804 Independent Call Girls Service ...
Russian Call Girls Chennai Madhuri 9907093804 Independent Call Girls Service ...Nehru place Escorts
 
Call Girls Service Chennai Jiya 7001305949 Independent Escort Service Chennai
Call Girls Service Chennai Jiya 7001305949 Independent Escort Service ChennaiCall Girls Service Chennai Jiya 7001305949 Independent Escort Service Chennai
Call Girls Service Chennai Jiya 7001305949 Independent Escort Service ChennaiNehru place Escorts
 
Call Girl Bangalore Nandini 7001305949 Independent Escort Service Bangalore
Call Girl Bangalore Nandini 7001305949 Independent Escort Service BangaloreCall Girl Bangalore Nandini 7001305949 Independent Escort Service Bangalore
Call Girl Bangalore Nandini 7001305949 Independent Escort Service Bangalorenarwatsonia7
 
Kesar Bagh Call Girl Price 9548273370 , Lucknow Call Girls Service
Kesar Bagh Call Girl Price 9548273370 , Lucknow Call Girls ServiceKesar Bagh Call Girl Price 9548273370 , Lucknow Call Girls Service
Kesar Bagh Call Girl Price 9548273370 , Lucknow Call Girls Servicemakika9823
 
Russian Call Girls Chickpet - 7001305949 Booking and charges genuine rate for...
Russian Call Girls Chickpet - 7001305949 Booking and charges genuine rate for...Russian Call Girls Chickpet - 7001305949 Booking and charges genuine rate for...
Russian Call Girls Chickpet - 7001305949 Booking and charges genuine rate for...narwatsonia7
 
Russian Call Girl Brookfield - 7001305949 Escorts Service 50% Off with Cash O...
Russian Call Girl Brookfield - 7001305949 Escorts Service 50% Off with Cash O...Russian Call Girl Brookfield - 7001305949 Escorts Service 50% Off with Cash O...
Russian Call Girl Brookfield - 7001305949 Escorts Service 50% Off with Cash O...narwatsonia7
 
Low Rate Call Girls Ambattur Anika 8250192130 Independent Escort Service Amba...
Low Rate Call Girls Ambattur Anika 8250192130 Independent Escort Service Amba...Low Rate Call Girls Ambattur Anika 8250192130 Independent Escort Service Amba...
Low Rate Call Girls Ambattur Anika 8250192130 Independent Escort Service Amba...narwatsonia7
 
CALL ON ➥9907093804 🔝 Call Girls Baramati ( Pune) Girls Service
CALL ON ➥9907093804 🔝 Call Girls Baramati ( Pune)  Girls ServiceCALL ON ➥9907093804 🔝 Call Girls Baramati ( Pune)  Girls Service
CALL ON ➥9907093804 🔝 Call Girls Baramati ( Pune) Girls ServiceMiss joya
 
Russian Call Girls in Chennai Pallavi 9907093804 Independent Call Girls Servi...
Russian Call Girls in Chennai Pallavi 9907093804 Independent Call Girls Servi...Russian Call Girls in Chennai Pallavi 9907093804 Independent Call Girls Servi...
Russian Call Girls in Chennai Pallavi 9907093804 Independent Call Girls Servi...Nehru place Escorts
 
Call Girls Service Noida Maya 9711199012 Independent Escort Service Noida
Call Girls Service Noida Maya 9711199012 Independent Escort Service NoidaCall Girls Service Noida Maya 9711199012 Independent Escort Service Noida
Call Girls Service Noida Maya 9711199012 Independent Escort Service NoidaPooja Gupta
 
Call Girls Colaba Mumbai ❤️ 9920874524 👈 Cash on Delivery
Call Girls Colaba Mumbai ❤️ 9920874524 👈 Cash on DeliveryCall Girls Colaba Mumbai ❤️ 9920874524 👈 Cash on Delivery
Call Girls Colaba Mumbai ❤️ 9920874524 👈 Cash on Deliverynehamumbai
 

Recently uploaded (20)

Bangalore Call Girls Majestic 📞 9907093804 High Profile Service 100% Safe
Bangalore Call Girls Majestic 📞 9907093804 High Profile Service 100% SafeBangalore Call Girls Majestic 📞 9907093804 High Profile Service 100% Safe
Bangalore Call Girls Majestic 📞 9907093804 High Profile Service 100% Safe
 
Call Girls In Andheri East Call 9920874524 Book Hot And Sexy Girls
Call Girls In Andheri East Call 9920874524 Book Hot And Sexy GirlsCall Girls In Andheri East Call 9920874524 Book Hot And Sexy Girls
Call Girls In Andheri East Call 9920874524 Book Hot And Sexy Girls
 
VIP Call Girls Pune Vani 9907093804 Short 1500 Night 6000 Best call girls Ser...
VIP Call Girls Pune Vani 9907093804 Short 1500 Night 6000 Best call girls Ser...VIP Call Girls Pune Vani 9907093804 Short 1500 Night 6000 Best call girls Ser...
VIP Call Girls Pune Vani 9907093804 Short 1500 Night 6000 Best call girls Ser...
 
Russian Call Girls in Bangalore Manisha 7001305949 Independent Escort Service...
Russian Call Girls in Bangalore Manisha 7001305949 Independent Escort Service...Russian Call Girls in Bangalore Manisha 7001305949 Independent Escort Service...
Russian Call Girls in Bangalore Manisha 7001305949 Independent Escort Service...
 
Call Girl Chennai Indira 9907093804 Independent Call Girls Service Chennai
Call Girl Chennai Indira 9907093804 Independent Call Girls Service ChennaiCall Girl Chennai Indira 9907093804 Independent Call Girls Service Chennai
Call Girl Chennai Indira 9907093804 Independent Call Girls Service Chennai
 
Call Girls Chennai Megha 9907093804 Independent Call Girls Service Chennai
Call Girls Chennai Megha 9907093804 Independent Call Girls Service ChennaiCall Girls Chennai Megha 9907093804 Independent Call Girls Service Chennai
Call Girls Chennai Megha 9907093804 Independent Call Girls Service Chennai
 
Call Girl Coimbatore Prisha☎️ 8250192130 Independent Escort Service Coimbatore
Call Girl Coimbatore Prisha☎️  8250192130 Independent Escort Service CoimbatoreCall Girl Coimbatore Prisha☎️  8250192130 Independent Escort Service Coimbatore
Call Girl Coimbatore Prisha☎️ 8250192130 Independent Escort Service Coimbatore
 
Call Girls Service In Shyam Nagar Whatsapp 8445551418 Independent Escort Service
Call Girls Service In Shyam Nagar Whatsapp 8445551418 Independent Escort ServiceCall Girls Service In Shyam Nagar Whatsapp 8445551418 Independent Escort Service
Call Girls Service In Shyam Nagar Whatsapp 8445551418 Independent Escort Service
 
Russian Call Girls Chennai Madhuri 9907093804 Independent Call Girls Service ...
Russian Call Girls Chennai Madhuri 9907093804 Independent Call Girls Service ...Russian Call Girls Chennai Madhuri 9907093804 Independent Call Girls Service ...
Russian Call Girls Chennai Madhuri 9907093804 Independent Call Girls Service ...
 
Call Girls Service Chennai Jiya 7001305949 Independent Escort Service Chennai
Call Girls Service Chennai Jiya 7001305949 Independent Escort Service ChennaiCall Girls Service Chennai Jiya 7001305949 Independent Escort Service Chennai
Call Girls Service Chennai Jiya 7001305949 Independent Escort Service Chennai
 
Call Girl Bangalore Nandini 7001305949 Independent Escort Service Bangalore
Call Girl Bangalore Nandini 7001305949 Independent Escort Service BangaloreCall Girl Bangalore Nandini 7001305949 Independent Escort Service Bangalore
Call Girl Bangalore Nandini 7001305949 Independent Escort Service Bangalore
 
Kesar Bagh Call Girl Price 9548273370 , Lucknow Call Girls Service
Kesar Bagh Call Girl Price 9548273370 , Lucknow Call Girls ServiceKesar Bagh Call Girl Price 9548273370 , Lucknow Call Girls Service
Kesar Bagh Call Girl Price 9548273370 , Lucknow Call Girls Service
 
Russian Call Girls Chickpet - 7001305949 Booking and charges genuine rate for...
Russian Call Girls Chickpet - 7001305949 Booking and charges genuine rate for...Russian Call Girls Chickpet - 7001305949 Booking and charges genuine rate for...
Russian Call Girls Chickpet - 7001305949 Booking and charges genuine rate for...
 
Russian Call Girl Brookfield - 7001305949 Escorts Service 50% Off with Cash O...
Russian Call Girl Brookfield - 7001305949 Escorts Service 50% Off with Cash O...Russian Call Girl Brookfield - 7001305949 Escorts Service 50% Off with Cash O...
Russian Call Girl Brookfield - 7001305949 Escorts Service 50% Off with Cash O...
 
Low Rate Call Girls Ambattur Anika 8250192130 Independent Escort Service Amba...
Low Rate Call Girls Ambattur Anika 8250192130 Independent Escort Service Amba...Low Rate Call Girls Ambattur Anika 8250192130 Independent Escort Service Amba...
Low Rate Call Girls Ambattur Anika 8250192130 Independent Escort Service Amba...
 
CALL ON ➥9907093804 🔝 Call Girls Baramati ( Pune) Girls Service
CALL ON ➥9907093804 🔝 Call Girls Baramati ( Pune)  Girls ServiceCALL ON ➥9907093804 🔝 Call Girls Baramati ( Pune)  Girls Service
CALL ON ➥9907093804 🔝 Call Girls Baramati ( Pune) Girls Service
 
Russian Call Girls in Chennai Pallavi 9907093804 Independent Call Girls Servi...
Russian Call Girls in Chennai Pallavi 9907093804 Independent Call Girls Servi...Russian Call Girls in Chennai Pallavi 9907093804 Independent Call Girls Servi...
Russian Call Girls in Chennai Pallavi 9907093804 Independent Call Girls Servi...
 
Call Girls Service Noida Maya 9711199012 Independent Escort Service Noida
Call Girls Service Noida Maya 9711199012 Independent Escort Service NoidaCall Girls Service Noida Maya 9711199012 Independent Escort Service Noida
Call Girls Service Noida Maya 9711199012 Independent Escort Service Noida
 
Escort Service Call Girls In Sarita Vihar,, 99530°56974 Delhi NCR
Escort Service Call Girls In Sarita Vihar,, 99530°56974 Delhi NCREscort Service Call Girls In Sarita Vihar,, 99530°56974 Delhi NCR
Escort Service Call Girls In Sarita Vihar,, 99530°56974 Delhi NCR
 
Call Girls Colaba Mumbai ❤️ 9920874524 👈 Cash on Delivery
Call Girls Colaba Mumbai ❤️ 9920874524 👈 Cash on DeliveryCall Girls Colaba Mumbai ❤️ 9920874524 👈 Cash on Delivery
Call Girls Colaba Mumbai ❤️ 9920874524 👈 Cash on Delivery
 

How Human Factors Testing Improves Healthcare Technology Procurement

  • 1. Using Human Factors in Hospital Technology Procurement Svetlena Taneva, PhD svetlena@humanfactors.ca Healthcare Human Factors University Health Network @healthcarehf @SvetlenaTaneva
  • 2. WHAT IS HUMAN FACTORS?
  • 3. Buxton, W. (2001). Less is More (More or Less), in P. Denning (Ed.). The Invisible Future: The seamless integration of technology in everyday life. New York: McGraw Hill, 145-179. What is Human Factors? The application of the science of human behaviour to the design of systems Engineering for human efficiency and effectiveness - accommodate the limits of the human user.
  • 4. Bridging the gap between human capability and demands of the task
  • 5.
  • 6. 210,000 - 400,000 US DEATHS annually due to preventable medical error James, J.T. (2013). A new, evidence-based estimate of patient harms associated with hospital care. Journal of Patient Safety, 9(3), 122-128
  • 7. 17,300 – 23,000 CANADIAN DEATHS annually due to preventable medical error CIHI (2016). Measuring Patient Harm in Canadian Hospitals. Canadian Patient Safety Institute. Baker, Norton, et al. (2004). The Canadian Adverse Events Study.
  • 8.
  • 10.
  • 11. The Opportunity Healthcare environments today: — Volume of technology increasing — Complexity of technology increasing — Interdependency of technologies increasing — Clinical staff busier — Increasingly concerned with errors associated with technology use
  • 12. Pitfalls of Traditional Procurement • Cost and functionality • Stakeholders: lack of front line users • Vendor demos: — Very slick — Biased by how well the vendor is liked — Biased by how well the vendor organizes the demo • Financial decisions are made early on • Financial review often carries as much weight as technical and clinical review
  • 13. HF goals in procurement — Objective evidence to inform the procurement decision — Identify risk and needed customizations to inform negotiations — Inform implementation strategy — Ensure best fit
  • 15. Usability Testing/Simulations Elements of a usability test: — Representative users — Representative environment (physical constraints) — Representative tasks with contender products — Realistic training, realistic use — Distractions — Record video and audio — Qualitative and quantitative — Semi-structured debrief
  • 18. Pump A Pump B Case Study: Large Volume Infusion Pumps
  • 19. Stakeholders — Clinical representation — Management — Infection control — Transportation — Supply and purchasing — Biomedical engineering
  • 20. HF Process List of prioritized tasks • Stakeholder input via task survey Shadowing builds context Draft scenarios • Clinicians validate and provide example pt. cases Refine based on heuristic evaluation Pilot and refine Usability Testing
  • 21. Usability Testing Protocol — Training — Setting up and programming the pump — Modifying the pump settings and hitting a soft limit — Modifying the pump settings and hitting a hard limit — Resolving an occlusion alarm — Resolving an empty reservoir alarm — Debrief and questionnaire
  • 22. Evaluation Criteria — Task accuracy & error frequency — Impact and severity of errors — Task efficiency — Number of requests for help — User feedback and preferences
  • 23. Categorization of Usability Issues Priority Definition High Issues that could have direct patient safety implications and/or prevent users from completing tasks Medium Issues that have a potential indirect patient safety impact and/or could result in considerable user frustration and inefficiencies Low Issues that affect general usability and/or the user’s overall impression of the system
  • 24. Number of Usability Issues Pump A Pump B
  • 25. Task Efficiency Number of Requests Pump A 18 Pump B 10 Average Time (mm:ss) Pump A 59:47 Pump B 56:35 Requests for guidance Time to complete scenarios
  • 26. 0 1 2 3 4 Easiest to use Best meets my clinical needs Safest Number of Participants Adult Acute Care - Exit Survey Cardinal Alaris Hospira Symbiq 0 1 2 3 4 Easiest to use Best meets my clinical needs Safest Number of Participants NICU - Exit Survey Cardinal Alaris Hospira Symbiq Questionnaires Pump A Pump B Pump A Pump B
  • 27. Overall Comparison Score Overall Comparison Score Pump A 10.70 Pump B 18.99
  • 29. HF Informed Procurement Examples Project • Electronic medical records Outcome Current products on the market are not easy to use, resulting in use-related patient safety risks. Decision to postpone purchase until better technology is available. HF value add Saved close to $1 billion and prevented introducing preventable risk.
  • 30. HF Informed Procurement Examples Project • Anesthesia Gas Machines Outcome Large variability in performance among the 4 products evaluated. Two products stood out as being significantly easier to use, with minimal patient safety implications. Investment made for the system that is both safe and best supports the specific organization's users and their workflows. HF value add
  • 31. HF Informed Procurement Examples Project • Surgical Instrument Management System Outcome Systems evaluated performed similarly across assessment criteria, but one of them demonstrated a significantly lesser number of critical patient safety and usability issues. Informed the customization negotiations, training requirements, mitigations and implementation strategy for the chosen product in order to prevent any potential use-safety risk. HF value add
  • 32. HF Informed Procurement Examples Project • ABG syringes Outcome Users were not able to distinguish the heparinized syringes from non- heparinized syringes. Packaging, plunger colour and labeling all contributed to safety errors, which perpetuated through supply stocking, feeds preparation and administration. The data was used to break the contract with the vendor. HF value add
  • 33. Benefits: Human Factors Informed Procurement — Objective evidence to inform the procurement decision — Identify risks and push for needed customizations — Improved adoption of technology — Improved efficiency — Decreased training — Improved patient safety
  • 34. Conclusions We cannot assume manufacturers test for safety Simulation takes into consideration the complexity of real healthcare environments Simulation uncovers system issues that may also be addressed through further simulation
  • 36.
  • 37.
  • 39.