This document provides an overview of standards and metrics for evaluating usability in healthcare. It discusses international standards that define usability and its key aspects of effectiveness, efficiency, and satisfaction. A new set of metrics is proposed to quantitatively measure these aspects of usability for medical devices. Effectiveness is measured by completeness, error, and assistance indices calculated for individual tasks and overall. Efficiency considers completion times. The metrics allow standardized usability testing and reporting to support design, risk analysis, and evaluation of medical devices.
Usability Validation Testing of Medical Devices and SoftwareUXPA Boston
The U.S. FDA and international regulatory bodies require usability testing of medical devices, products, software, and systems as part of their overall validation. Manufacturers must demonstrate that all potential use-related hazards have been identified, prioritized, and mitigated. The method for demonstrating this is human factors/usability engineering (HF/UE) validation testing. However, the way we conduct these studies is in many ways different from the way we conduct studies of non-medical products and systems.
This topic is relevant to the Boston UX community given the convergence of consumer and medical devices, as well as the rise of wearable technologies and the apps that interact with them. This presentation will cover the key aspects of HF/UE validation (a.k.a. ‘summative’) testing and what the FDA expects in the final HF/UE summary report.
Importantly, this session will consist of half presentation and half Q&A, with the audience driving the discussion toward current issues, questions, and challenges that are relevant to them.
Making India to a global healthcare hub, it is not only about bringing new technology but also we have to take care of the
existing technology. The healthcare hub is the leading factor for current economic growth of India. Human Factor Engineering
(HFE) plays a vital role in this field. In medical or healthcare, the field is named as Medical Human Factor Engineering (MHFE).
This paper discusses on how MHFE responsible for strengthen the Technology Management of Hospital, Hazards from device
failure and use related, Human Factors consideration in medical device use and case study on (Infusion Pumps) errors committed by
users in each clinical area. Now the challenging issue for HFE is to design a proper workspace to avoid human errors and the four
workspace design principles of Sanders & McCormick (1993) is also discussed. This paper deals with the Computer-aided-design
(CAD) systems and a failure mode and effects analysis (FMEA) technique with Simple Organizational Structure of HFE in designing the workspace.
7 Components to Medical Device Usability Testing SuccessMargee Moore
Despite the publication of various relevant guidance on medical device usability testing and standards for human factors testing confusion regarding best practices still exists. This presentation provides clear language and seven components for planning successful usability testing for medical device development.
Regulatory and Quality Affairs: Answers to FDA and ISO Gray AreasApril Bright
Every day, people like you in companies everywhere are sidetracked from more pressing priorities by questions and scenarios that aren’t clearly explained in a regulation or standard (a.k.a. "gray areas"). This panel of regulatory and quality experts were charged with mitigating your roadblocks and getting you on your way. Our panel shared their perspective on the pressing questions received from a pre-conference attendee questionnaire, including UDI and supplier relationships.
Usability Validation Testing of Medical Devices and SoftwareUXPA Boston
The U.S. FDA and international regulatory bodies require usability testing of medical devices, products, software, and systems as part of their overall validation. Manufacturers must demonstrate that all potential use-related hazards have been identified, prioritized, and mitigated. The method for demonstrating this is human factors/usability engineering (HF/UE) validation testing. However, the way we conduct these studies is in many ways different from the way we conduct studies of non-medical products and systems.
This topic is relevant to the Boston UX community given the convergence of consumer and medical devices, as well as the rise of wearable technologies and the apps that interact with them. This presentation will cover the key aspects of HF/UE validation (a.k.a. ‘summative’) testing and what the FDA expects in the final HF/UE summary report.
Importantly, this session will consist of half presentation and half Q&A, with the audience driving the discussion toward current issues, questions, and challenges that are relevant to them.
Making India to a global healthcare hub, it is not only about bringing new technology but also we have to take care of the
existing technology. The healthcare hub is the leading factor for current economic growth of India. Human Factor Engineering
(HFE) plays a vital role in this field. In medical or healthcare, the field is named as Medical Human Factor Engineering (MHFE).
This paper discusses on how MHFE responsible for strengthen the Technology Management of Hospital, Hazards from device
failure and use related, Human Factors consideration in medical device use and case study on (Infusion Pumps) errors committed by
users in each clinical area. Now the challenging issue for HFE is to design a proper workspace to avoid human errors and the four
workspace design principles of Sanders & McCormick (1993) is also discussed. This paper deals with the Computer-aided-design
(CAD) systems and a failure mode and effects analysis (FMEA) technique with Simple Organizational Structure of HFE in designing the workspace.
7 Components to Medical Device Usability Testing SuccessMargee Moore
Despite the publication of various relevant guidance on medical device usability testing and standards for human factors testing confusion regarding best practices still exists. This presentation provides clear language and seven components for planning successful usability testing for medical device development.
Regulatory and Quality Affairs: Answers to FDA and ISO Gray AreasApril Bright
Every day, people like you in companies everywhere are sidetracked from more pressing priorities by questions and scenarios that aren’t clearly explained in a regulation or standard (a.k.a. "gray areas"). This panel of regulatory and quality experts were charged with mitigating your roadblocks and getting you on your way. Our panel shared their perspective on the pressing questions received from a pre-conference attendee questionnaire, including UDI and supplier relationships.
Organisational learning in industrial companies – Experience from GermanyDoytchin Doytchev
Describes how organisations learn from past accidents and incidents and what changes they make to procedures , processes and equipment to enhance safety.
Conference presentation.
This report examines the industry perception of the EDC and eCOA/ePRO markets based on the responses of over 150 industry professionals experienced with these systems. ISR has asked respondents how their respective companies handle the decisions associated with outsourcing these systems as well as how they would prefer the outsourcing be handled. Additionally, ISR has benchmarked 22 EDC and 22 eCOA/ePRO systems based on 651 EDC and 432 ePRO service provider encounters. Each provider’s performance is evaluated and benchmarked based on the performance of these providers against expectations across 18-21 attributes.
Learn more: http://www.isrreports.com/product/edc-and-ecoaepro-market-dynamics-and-service-provider-performance-2015/
OSMA: Orthopedic Industry's Top Regulatory Challenges and OpportunitiesApril Bright
The Orthopaedic Surgical Manufacturers Association, a collective voice of orthopedic device companies that influences the decision of worldwide regulatory agencies and standards bodies, will highlight the main regulatory changes impacting the industry. This session is for any orthopedic professional who wants a forecast of regulatory pressures and seeks direction on how to shape change. Attendees will learn how FDA, European agencies and IMDRF are approaching harmonization and alignment of standards, regulations and guidance. OSMA Members will provide future trends and opportunities afforded via FDA’s National Evaluation System for Health Technology (NEST), facilitation of innovation through partnerships and global harmonization of regulatory submissions and facility assessments.
Engineering Research Publication
Best International Journals, High Impact Journals,
International Journal of Engineering & Technical Research
ISSN : 2321-0869 (O) 2454-4698 (P)
www.erpublication.org
Slides presented at "Getting Your Medical Device FDA Approved" event, presented by Mentor Graphics Embedded Software, discussing how to address the enhanced scrutiny from government agencies that can introduce significant delays with the commercial release of software-related medical devices.
Getting through the FDA review as quickly as possible requires a clear understanding of the development standards, documentation and testing that is now expected for Medical devices. During this session we discussed how FDA hot buttons affect your medical device submission will be discussed, including:
-Requirements for software development as outlined in IEC 62304
-Content considerations for premarket submissions
-Human factors engineering as a platform for enhanced user safety
-Provisions for data security and protection against unauthorized wireless access
We reviewed the design control requirements and product development approach that can shorten your medical device's path to market with a focus on safety, human factors engineering and security.
Operations: Top Reasons for Long Lead Times and What to Do About ThemApril Bright
Long lead times remain one of the most vocalized challenges that orthopedic manufacturers face today. Customers, profits, plans and personnel are all negatively impacted by them. James Kwan has worked on the OEM and the supplier sides of orthopedics, and shared his ideas and successful experiences to help you optimally respond to lead times, reduce them and ultimately create and sustain an agile supply chain.
Leverage Computational Modeling and Simulation for Device Design - OMTEC 2017April Bright
Computational modeling and simulation (CM&S) has the potential to revolutionize medical devices by accelerating innovation and providing comprehensive evidence of long-term safety. For example, CM&S can provide performance benchmarks, assess design parameter interdependencies, evaluate a variety of use conditions, provide visualization of complex processes and become a core element of device submissions and approvals. This presentation will begin with an overview of the use of CM&S throughout the orthopaedic implant lifecycle, followed by a review of the current regulatory direction regarding the use of CM&S in device submissions. Next, a series of case studies based on a variety of orthopaedic implants will demonstrate the application of CM&S at various phases of the product lifecycle in more detail. The examples will also highlight the effects of modeling assumptions on model credibility and some verification and validation best practices.
This presentation will position CM&S as a credible and common means for device companies and FDA to demonstrate the safety of medical devices, and thereby ensure safety, reduce cost and accelerate the pathway toward “first in the world” access to products in the U.S.
A Neural Network Based Diagnostic System for Classification of Industrial Car...CSCJournals
Even with many years of research efforts, the occupational exposure limits of different risk factors for development of Musculoskeletal disorders (MSDs) have not yet been established. One of the main problems in setting such guidelines is the limited understanding of how different risk factors of MSDs interact in causing the injury, as the nature and mechanism of these disorders are relatively unknown phenomena. The task of an industrial ergonomist is complicated because the potential risk factors that may contribute to the onset of the MSDs interact in a complex way, and require an analyst to apply elaborate data measurement and collection techniques for a realistic job analysis. This makes it difficult to discriminate well between the jobs that place workers at high or low risk of above disorders. The main objective of this study was to to develop an artificial neural network based diagnostic system which can classify industrial jobs according to the potential risk for physiological stressors due to workplace design. Such a system could be useful in hazard analysis and injury prevention due to manual handling of loads in industrial environments. The results showed that the developed diagnostic system can successfully classify jobs into low and high risk categories of above musculoskeletal disorders based on carrying task characteristics. The Neural network based system developed gave the correct classification of the analysed industrial jobs with low and high risk. So, the system can be used as an expert system which, when properly trained, will classify carrying load by male and female workers into two categories of low risk and high risk work, based on the available characteristics factors.
Surgeon Perspective on Additive Manufacturing April Bright
As additive manufacturing gains popularity as a production process, it’s important to understand the perspective of surgeons engaged with the technology. Nirav Shah, M.D., has consulted with device companies on additively-manufactured implants and is involved in the research of 3D-printed tissue. He will share his perspective and research on the current state of additive manufacturing in orthopaedics, and provide ideas on how the technology could be used in the future.
In this short talk I’ll cover the term ‘inside attacker’. What does it mean and what are the stats? Are inside attacks getting worse or better? If worse, what can we do about that?
Presented at DevOpsDays Kansas City, Sept 21 2017.
Organisational learning in industrial companies – Experience from GermanyDoytchin Doytchev
Describes how organisations learn from past accidents and incidents and what changes they make to procedures , processes and equipment to enhance safety.
Conference presentation.
This report examines the industry perception of the EDC and eCOA/ePRO markets based on the responses of over 150 industry professionals experienced with these systems. ISR has asked respondents how their respective companies handle the decisions associated with outsourcing these systems as well as how they would prefer the outsourcing be handled. Additionally, ISR has benchmarked 22 EDC and 22 eCOA/ePRO systems based on 651 EDC and 432 ePRO service provider encounters. Each provider’s performance is evaluated and benchmarked based on the performance of these providers against expectations across 18-21 attributes.
Learn more: http://www.isrreports.com/product/edc-and-ecoaepro-market-dynamics-and-service-provider-performance-2015/
OSMA: Orthopedic Industry's Top Regulatory Challenges and OpportunitiesApril Bright
The Orthopaedic Surgical Manufacturers Association, a collective voice of orthopedic device companies that influences the decision of worldwide regulatory agencies and standards bodies, will highlight the main regulatory changes impacting the industry. This session is for any orthopedic professional who wants a forecast of regulatory pressures and seeks direction on how to shape change. Attendees will learn how FDA, European agencies and IMDRF are approaching harmonization and alignment of standards, regulations and guidance. OSMA Members will provide future trends and opportunities afforded via FDA’s National Evaluation System for Health Technology (NEST), facilitation of innovation through partnerships and global harmonization of regulatory submissions and facility assessments.
Engineering Research Publication
Best International Journals, High Impact Journals,
International Journal of Engineering & Technical Research
ISSN : 2321-0869 (O) 2454-4698 (P)
www.erpublication.org
Slides presented at "Getting Your Medical Device FDA Approved" event, presented by Mentor Graphics Embedded Software, discussing how to address the enhanced scrutiny from government agencies that can introduce significant delays with the commercial release of software-related medical devices.
Getting through the FDA review as quickly as possible requires a clear understanding of the development standards, documentation and testing that is now expected for Medical devices. During this session we discussed how FDA hot buttons affect your medical device submission will be discussed, including:
-Requirements for software development as outlined in IEC 62304
-Content considerations for premarket submissions
-Human factors engineering as a platform for enhanced user safety
-Provisions for data security and protection against unauthorized wireless access
We reviewed the design control requirements and product development approach that can shorten your medical device's path to market with a focus on safety, human factors engineering and security.
Operations: Top Reasons for Long Lead Times and What to Do About ThemApril Bright
Long lead times remain one of the most vocalized challenges that orthopedic manufacturers face today. Customers, profits, plans and personnel are all negatively impacted by them. James Kwan has worked on the OEM and the supplier sides of orthopedics, and shared his ideas and successful experiences to help you optimally respond to lead times, reduce them and ultimately create and sustain an agile supply chain.
Leverage Computational Modeling and Simulation for Device Design - OMTEC 2017April Bright
Computational modeling and simulation (CM&S) has the potential to revolutionize medical devices by accelerating innovation and providing comprehensive evidence of long-term safety. For example, CM&S can provide performance benchmarks, assess design parameter interdependencies, evaluate a variety of use conditions, provide visualization of complex processes and become a core element of device submissions and approvals. This presentation will begin with an overview of the use of CM&S throughout the orthopaedic implant lifecycle, followed by a review of the current regulatory direction regarding the use of CM&S in device submissions. Next, a series of case studies based on a variety of orthopaedic implants will demonstrate the application of CM&S at various phases of the product lifecycle in more detail. The examples will also highlight the effects of modeling assumptions on model credibility and some verification and validation best practices.
This presentation will position CM&S as a credible and common means for device companies and FDA to demonstrate the safety of medical devices, and thereby ensure safety, reduce cost and accelerate the pathway toward “first in the world” access to products in the U.S.
A Neural Network Based Diagnostic System for Classification of Industrial Car...CSCJournals
Even with many years of research efforts, the occupational exposure limits of different risk factors for development of Musculoskeletal disorders (MSDs) have not yet been established. One of the main problems in setting such guidelines is the limited understanding of how different risk factors of MSDs interact in causing the injury, as the nature and mechanism of these disorders are relatively unknown phenomena. The task of an industrial ergonomist is complicated because the potential risk factors that may contribute to the onset of the MSDs interact in a complex way, and require an analyst to apply elaborate data measurement and collection techniques for a realistic job analysis. This makes it difficult to discriminate well between the jobs that place workers at high or low risk of above disorders. The main objective of this study was to to develop an artificial neural network based diagnostic system which can classify industrial jobs according to the potential risk for physiological stressors due to workplace design. Such a system could be useful in hazard analysis and injury prevention due to manual handling of loads in industrial environments. The results showed that the developed diagnostic system can successfully classify jobs into low and high risk categories of above musculoskeletal disorders based on carrying task characteristics. The Neural network based system developed gave the correct classification of the analysed industrial jobs with low and high risk. So, the system can be used as an expert system which, when properly trained, will classify carrying load by male and female workers into two categories of low risk and high risk work, based on the available characteristics factors.
Surgeon Perspective on Additive Manufacturing April Bright
As additive manufacturing gains popularity as a production process, it’s important to understand the perspective of surgeons engaged with the technology. Nirav Shah, M.D., has consulted with device companies on additively-manufactured implants and is involved in the research of 3D-printed tissue. He will share his perspective and research on the current state of additive manufacturing in orthopaedics, and provide ideas on how the technology could be used in the future.
In this short talk I’ll cover the term ‘inside attacker’. What does it mean and what are the stats? Are inside attacks getting worse or better? If worse, what can we do about that?
Presented at DevOpsDays Kansas City, Sept 21 2017.
Risk Management in IES 60601. Medical Devices, Creation and content of RMF, Methods for the visualization and identification of harms and hazards,
Creating a RMF – Minimal Documentation,
Common errors when creating a RMF.
Shahid, M., & Kamran, F. (2015). Causal Relationship between Macroeconomic Factors and Stock Prices in Pakistan. International Journal Of Management And Commerce Innovations, 3(2), 172-178. Retrieved from http://researchpublish.com/journal/IJMCI/Issue-2-October-2015-March-2016/0
Human error and secure systems - DevOpsDays Ohio 2015Dustin Collins
We see reports all the time with headlines like "90% of data breaches caused by human error". But what does that really mean? In this talk I will cover the traditional view of human error and how it hinders our ability to develop and maintain secure systems. Other industries have improved safety and security by shifting their view of human error. We can apply many of the the same concepts to software development and operations, minimizing risk and maximizing learning opportunity.
Clinical Simulation as an Evaluation Method in Health Inf.docxbartholomeocoombs
Clinical Simulation as an Evaluation Method
in Health Informatics
Sanne JENSEN
a,1
a
The Capital Region of Denmark, Copenhagen, Denmark
Abstract. Safe work processes and information systems are vital in health care.
Methods for design of health IT focusing on patient safety are one of many
initiatives trying to prevent adverse events. Possible patient safety hazards need to
be investigated before health IT is integrated with local clinical work practice
including other technology and organizational structure. Clinical simulation is
ideal for proactive evaluation of new technology for clinical work practice.
Clinical simulations involve real end-users as they simulate the use of technology
in realistic environments performing realistic tasks. Clinical simulation study
assesses effects on clinical workflow and enables identification and evaluation of
patient safety hazards before implementation at a hospital. Clinical simulation also
offers an opportunity to create a space in which healthcare professionals working
in different locations or sectors can meet and exchange knowledge about work
practices and requirement needs. This contribution will discuss benefits and
challenges of using clinical simulation, and will describe how clinical simulation
fits into classical usability studies, how patient safety may benefit by use of
clinical simulation, and it will describe the different steps of how to conduct
clinical simulation. Furthermore a case study is presented.
Keywords. Ergonomics, eHealth, qualitative evaluation, clinical simulation, risk,
safety.
1. Introduction
Implementation of health IT in relation to improvement of patient safety and
optimization of work flow is a paradox [1]. Even though health IT is intended and
anticipated to have a positive impact on quality and efficiency of health care [2], the
application of new technology in healthcare may also increase patient safety hazards [3,
4]. Studies show that adverse events are indeed often related to the use of technology
[5-7].
Design of health IT focusing on protecting patient safety is one of many initiatives
trying to prevent adverse events [8, 9].
2
Patient safety does not entirely rely on
technology but is highly influenced by the interaction between users and technology in
a specific context [10], and sociotechnical issues and human factors are related to many
unintended consequences and patient safety hazards [7, 8, 11]. Possible patient safety
hazards such as design of the IT system itself; embedding of IT system into local work
1
Corresponding author: Sanne Jensen, The Capital Region of Denmark, Borgervanget 7, 2100
Copenhagen O, Denmark, [email protected]
2
See also: F. Magrabi et al., Health IT for patient safety and improving the safety of health IT, in: E.
Ammenwerth, M. Rigby (eds.), Evidence-Based Health Informatics, Stud Health .
Designing medical devices that meet the needs of patients, healthcare providers, and the broader healthcare system requires a thorough understanding of human factors and usability principles. In this presentation, we'll explore the application of these principles to medical device design.
OVERVIEW OF CRITICAL FACTORS AFFECTING MEDICAL USER INTERFACES IN INTENSIVE C...hiij
This paper provides a comprehensive overview of critical factors, which affect on-screen user interfaces of
medical devices in Intensive Care Unit (ICU). A literature survey with relevant research publications has
led to selection of thirty eight critical factors in ICU. The critical factors identified are categorized into
various groups based on three major aspects – system evaluation parameters, constituents of patient
management and user interface design. Physicians’ survey, in which five physicians are involved, is used to
categorize the identified critical factors into related groups. In the process, fourteen critical factors are
mainly selected, which affect on-screen user interface design of medical devices. The applicability of such
factors is demonstrated with the help of a case study of head-injury patient admitted in ICU. The critical
factors identified are definitely useful to device manufacturers, user interface designers, ICU
administrators and physicians for improved device design, ICU resource management and patient care.
Pier Angelo Sottile, Chairman of the Technical Committee UNINFO dealing with eHealth, presents standardization activities going on in UNI, CEN and ISO about health and applications (APPs)
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
2. 10
Figure 1. Methodological steps followed for the usability testing review.
MATERIAL AND METHODS
Usability and ergonomics are often misunderstood and
considered synonymous when designing and evaluating
products. Ergonomics (EN ISO 6385-2004) represents
the scientific field dealing with the general optimization of
a system (e.g. interactions between human and non-
human elements of a system and/or principles, data and
methods to design systems aimed at human well-being)
while usability (and usability engineering) focuses on
safety issues with reference to users, tasks and context
of use.
As shown in figure 1, in order to provide technical
guidance on usability testing a systematic review of
literature, international regulations and experts' opinions
(users' and manufacturers') was carried out by
addressing the following key questions:
Have any usability measurable indicators been
defined? If so, which ones?
Is any mathematical formula available to
calculate each specific usability index and the general
one?
What setting/location would be appropriate for
carrying out usability tests? Why?
INTERNATIONAL STANDARDS ON USABILITY IN
HEALTHCARE
The historic development of regulations about usability is
shown in Figure 2.
The standard upon which the concept of usability was
built is ISO 9241-11, in which usability and the
procedures to measure it were defined for the first time.
The American technical regulations applied the concept
of usability to healthcare in 2001. ANSI-AAMI HE 74
represents one of the main foundations of the EN 62366
European Standard (Italian CEI EN 62366), which was
published in October 2010 and dealt with usability test of
medical devices, and of the three editions of the EN
60601-1-6 European Standard, which were
acknowledged by three editions of the Italian Standard
CEI EN 60601-1-6 in 2006, 2008 and 2011 respectively.
Finally, the need to communicate the results of usability
tests in an effective and standardized way lead to the
development of the ISO/IEC 25062 International
Standard, which states how a Usability Report is to be
organized and which features it has to have.
ISO 9241-11:1998
The International Standard on “Ergonomic requirements
for office work with visual display terminals (VDTs) –
Guidance on Usability” defines usability as “the extent to
which a product can be used by specified users to
achieve specified goals with effectiveness, efficiency and
satisfaction in a specified context of use.”
Indeed, usability is based on three basic elements
(Figure 3):
Effectiveness: level of accuracy and completeness in
carrying out the functions the device is meant to perform;
Efficiency: effectiveness in relation to the resources used.
Time efficiency is related to the time needed to carry out
the functions of the device. Other types of efficiency exist,
such as economic efficiency (in relation to costs) and
human efficiency (in relation to human resources).
3. 11
Figure 1. Historic development of national and international regulations about the concept of Usability.
Figure 2. Usability Framework according to ISO 9241‐11:1998 - Guidance on usability.
User Satisfaction: synergy of information obtained from
the user both through behavioural analyses, interviews
and questionnaires administered before, during and after
a usability test.
As reported in the framework shown in Figure 3, a
usability test makes sense only if it is set in a specific
context of use, which consists of users, tasks, equipment
(hardware, software and materials) and of the physical
4. 12
and social environment in which the test is carried out. All
of the above mentioned elements can influence the
usability of a product in a work system (ISO 9241-
11:1998).
Here follow the four main application fields of usability
tests:
Comparative Analysis of different products or of
different versions of the same product;
Support in designing a product;
Diagnostic Evaluation applied to identify specific
elements responsible of usability problems of a device;
Planning specific and appropriate training about
how to use a device.
The ISO 9241-11:1998 Standard also defines the most
appropriate tools to measure the elements to be
evaluated through a usability test and states the need to
develop and apply clear and measurable quantitative and
semi-quantitative indexes. Moreover, this standard
underlines the importance of having a simulation
laboratory available, where one can arrange different
scenarios, simulate several contexts of use and have
better control over the variables present while using a
device (Daniels et al., 2007).
CEI EN 62366:2008
Application of usability engineering to medical
devices.
The IEC62366:2007 standard acknowledged by Italian
regulations in the CEI EN 62366:2008 standard identifies
poor usability of medical devices as one of the major
causes of use errors, because it is closely linked to poor
ease of use and learnability. Moreover, the application
field of medical devices proves to be highly critical
because of the increased technical complexity of devices
and of their availability to users who lack any healthcare
training, such as patients themselves.
With respect to ISO 9241-11:1998 Standard, two major
innovations have been introduced by the American
Standard ANSI/AAMI HE 74:2001first, and by the
European Standard later. The first innovation consists in
the introduction of the concept of learnability within the
definition of usability. The second one concerns the
definition of “primary operating functions”, which are the
functions to be taken into consideration when one carries
out a usability test. The definition of primary operating
functions includes both the functions which are frequently
used and those which are critical in relation to safety.
The concept of usability is considered of primary
importance in relation to safety, because of the the link
between usability test and the process of risk analysis
applied to medical devices, as it is described in the EN
ISO 14971 Standard which bears the title “Application of
risk management to medical devices”. The connections
concern the identification of risk elements in using the
device according to the manufacturer’s intended use, the
identification of risks and the implementation and
validation of procedures and actions aiming at reducing
risks.
ANSI/AAMI HE 74:2001
Human factor design process for medical devices
Attachment D of EN 62366 Standard clearly draws on the
ANSI/AAMI HE 74:2001 American Standard, which
defines the fields of application of usability test in the
steps involved in designing a device:
Conceptual Design, definition of aims and of the user’s
need;
Definition of technical requirements and technical
solutions;
Complete and detailed implementation of the technical
solutions chosen;
Test on the prototype
CEI EN 60601-1-6:2011
Medical electrical equipment – Part 1: General
requirements for basic safety and essential
performance – Collateral standard: Usability
The EN 60601-1-6:2010-04 European standard,
acknowledged by Italian regulations in the CEI EN
60601-1-6:2011-05 standard replaces the second edition
of IEC 60601-1-6 and aligns with the Usability
Engineering Process described in the IEC 62366
standard. This is due to the fact that, as healthcare
evolves, less skilled operators, including patients
themselves, are now using medical electrical equipment,
while the medical electrical equipment itself is becoming
more complicated (IEC 60601-1-6).
The above mentioned standard provides some important
innovations with respect to previous editions for what
concerns the following themes:
The so-called “primary operating functions” consider
correct and safe performances of the device by including
the frequently used functions and those functions related
to the safety of a medical device;
In order to carry out an exhaustive usability test to assess
the basic safety and essential performance of a device,
5. the EN 60601-1-6 standard suggests to take into
consideration the reasonably foreseeable user’s misuse,
besides the correct use intended by the producer.
Manufacturers shall not consider any form of incorrect
use. Moreover, the usability engineering process
concerns risk identification but not mitigation associated
with abnormal use.
The term “patient” shall include animals as well.
Accompanying documents (instruction for use and
technical description) have to be included in the usability
engineering process as part of operator-equipment
interface.
Worst case and frequent use scenarios shall be included
in the usability engineering process.
The standard underlines once again the basic concept
that usability is to be assessed within the context of use
of a device (Figure 4) taking into consideration the type of
user, the specific device and its intended use.
Finally, the standard contains a list of the main
applications of usability tests:
Support to product design;
Support to product prototyping;
Planning of workload;
Cost-benefit analysis;
Risk analysis;
Comparative analysis between different devices
or different versions of the same device.
ISO/IEC 25062
Software engineering – Software product Quality
Requirements and Evaluation (SQuaRE) – Common
Industry Format (CIF) for usability test reports.
The significance of this standard lies in the definition of
one standard format to present in a clear and effective
way the results of a usability test carried out on a device
and/or on a method to evaluate the usability of a device.
The way such results are communicated is of primary
importance, because significant decisions and choices
can depend on this. Both the IEC 62366:2007 and IEC
60601-1-6:2010 standards refer to the ISO/IEC 25062
standard to report on the measures obtained in a usability
test and include them in a usability engineering file. Here
follow the elements which are to be included in a usability
test:
Participants in the test must represent the real user
population for which the device is intended;
Primary functions are to be defined;
Measures related to effectiveness, efficiency and user
satisfaction are to be defined and included in the report.
13
RESULTS
On the basis of the instructions contained in the set of
standards analysed above, here are described the
proposed new metrics which have been developed to
measure the characteristics of usability: effectiveness,
efficiency, user satisfaction and learnability.
Effectiveness
The effectiveness index of a device is the result of the
combination of three components: the completeness of
the tasks carried out with success by the participants, the
number of errors made and the number of assists needed
by the participants to carry out the tasks. The
effectiveness index can be calculated both in relation to
the overall performance of the device and to a specific
primary function. In detail, the effectiveness index per
task is composed of the following indicators:
The Completeness Index per Specific Task-i (ICSi)
results from the number of participants who have
completed the specific task-i with success divided by the
number of participants, while the General Completeness
Index (ICS) is the level of completeness which
characterizes a device in all of the tasks tested by the
participants and it is defined as the sum of the
completeness indexes ICSi divided by the number of
tasks, as sown in equation (1).
[ICS=(∑ICSi)/number of tasks] (1)
The Error Index per specific task-i (IEGi) measures the
percentage of mistakes which occur while participants
use the device in each specific task and it is defined as
the total number of errors per task-i divided by the
number of participants, while the General Error Index
(IEG) provides information about mistakes with relation to
the general use of the device and it is defined as the sum
of the error indexes per task-i divided by the number of
tasks, as shown in equation (2).
[IEG = (∑IEGi)/number of tasks] (2)
The Assist Index per specific task-I (IAGi) results from the
total number of assists per task-i divided by the number
of participants, while the General Assist Index (IAG) is
the sum of the assist indexes per task-i divided by the
number of tasks, as shown in equation (3).
[IAG = (∑IAGi)/number of tasks] (3)
The General Effectiveness Index per task-i (IGEi) is the
synthetic index which represents the general
effectiveness of the device for a specific task and it is
6. 14
Figure 4. Usability engineering process
Figure 5. Check-list used for data collection by the evaluators during a usability test.
defined as the linear combination between IAGi, IEGi and
ICSi, as shown in equation (4), while the General
Effectiveness Index (IGE) results from the linear
combination between IAG, IEG and ICS, as shown in
equation (5).
[IGEi=(3*ICSi - 2*IEGi – IAGi )/6] (4)
[IGE =(3*ICS - 2*IEG – IAG )/6] (5)
Finally, figure 5 shows the specific check-list used to
collect data during the test. Thanks to this check-list, it is
possible to know if each task was completed, how many
mistakes were made and how many assists were given to
the participants. Such information is available for each
task.
Efficiency
The efficiency Index can be evaluated as the result of the
combination between the effectiveness and the
completion times achieved by the participants in the test.
The efficiency index can be calculated both in relation to
the overall performance of the device and for a specific
primary function or a single task.
In detail, the efficiency index per task is composed of the
following indicators:
Time Efficiency index per specific task-i (IETi) results
from the number of participants who obtained a
completion time higher than the value expected by
experts divided by the number of participants who
7. completed the task. The Completion Time can be
calculated per specific task (TCi) and results from the
sum of the completion times per participant divided by the
number of participants, while the General Completion
Time (TC) of a device is the sum of the TCi divided by the
number of tasks, as shown in equation (6).
[TC= (∑TCi)/number of tasks] (6)
It is important to underline that in the case of a usability
test carried out on one device only involving one
homogeneous group composed of several participants,
the Expected Completion time per specific task-i (TCi) is
provided by field experts. In the case of an analysis
carried out on two devices or two versions of the same
device or involving two groups of participants, the
significance range is calculated through a t-student test.
Finally, in the case of a usability test carried out on one
device with more than two groups of participants involved
or on more than two devices or versions of a device with
one group of participants involved, the significance range
is calculated through an ANOVA test. The check-list
used for the time collection is shown in figure 4, in the
areas “starting time” and “ending time.”
User Satisfaction and Learnability
User satisfaction and learnability can be assesses by
administering self-report tools, such as interviews and
questionnaires, during various phases of a usability test.
Literature about the usability of websites and information
systems includes several validated questionnaires which
aim at evaluating subjective features of the operator’s
use experience. The purpose of some questionnaires is
to gather information about the participants’ general
opinion about the usability of a system, while other
questionnaires aim at evaluating user satisfaction and
learnability. Here follow some of the issues surveyed by
the above mentioned tools: user’s previous experience
(Shneiderman, 1987), ease of use of the device (Lund,
2001), learnability (Kirakowsk et al. 1992, Shneiderman
1987 and Lund 2001), perceived efficiency (Kirakowsk et
al. 1992), perceived control on the device (Kirakowsk et
al. 1992), use satisfaction (Lewis 1991 and Lund 2001),
emotional reactions (Kirakowsk et al. 1992, Isomursu
2007) and user’s expectations (Albert and Dixon 2003, ,
Thayer and Dugan 2009). Unlike what happens in the
field of usability of websites and information systems, no
validated questionnaires are known which aim at
assessing subjective features of users’ experience with
medical devices. A few authors (Chiu et al.2004,
Follmann et al. 2010, Garmer et al. 2002, Hersch et al.
2009)) used ad hoc built questionnaires, which were
administered at the end of usability tests with the purpose
to assess both the
15
participants’ subjective perception about the usability of
the electro medical devices tested and, more specifically,
the users’ satisfaction and learnability of such devices. In
order to assess the user’s general satisfaction with
electromedical information devices (electronic medical
record system), other authors [Jaspers et al. 2008, Sitting
et al. 1999), used the Questionnaire for User Interaction
Satisfaction (QUIS) (Chin et al. 1988), which was
originally intended to be applied to information systems.
For what concerns the timing and phases to administer
the questionnaires, there are three possibilities:
Pre-test phase: participants are administered
questionnaires before the usability test begins. Thayer
and Dugan (Thayer and Dugan 2009) defined the
questionnaires administered in this phase as ”pre-context
questionnaires”, while Rubin and Chisnell (Rubin, J. and
Chisnell, 2008) refer to this kind of tool as “background
questionnaires”. Such questionnaires aim at gathering
information about the participants’ past experience, which
is helpful to better understand their behaviour and
performance during the test. This kind of questionnaires
is composed of a series of items which survey the
subjects’ experiences, attitudes and preferences in the
fields which could affect their performance. The above
mentioned authors state that these questionnaires are
useful to check whether the subjects recruited are
appropriate for the test.
Test phase: participants are administered questionnaires
before and/or after each task. Dumas and Redish
(Dumas and Redish, 1999) suggested administering
short interviews or questionnaires in this phase and to
have participants express their answers using Likert
scales. Lewis (Lewis 1991) put together a questionnaire
(After Scenario Questionnaire – ASQ) composed of
three questions to be asked at the end of each scenario.
Participants are required to answer using a 1-5 Likert
scale, with a range which varies from strongly disagree to
strongly agree, which aims at assessing the participants’
self-referred satisfaction about the tasks they carried out.
Sauro and Dumas (Sauro and Dumas 2009) stated that
carrying out an evaluation at the end of each scenario
has the advantage of providing more diagnostic
information about usability and more valid measures.
Albert and Dixon (Albert and Dixon 2003) applied a
different procedure, which implies asking two questions,
that is, one before and one just after each task is carried
out, in order to assess both the participants’ expectations
and experience as for the ease of use of the device. By
comparing such data, useful information can be gathered
and, if needed, actions can be planned to improve or
correct certain features of the device.
Finally, information about subjective experience can be
gathered by using the thinking-aloud technique, that
8. 16
Figure 6. Conceptual connection of usability to safety and risk management for medical devices (EN 62366, EN 14971).
consists in asking participants to express their thoughts in
words while they are carrying out a task. Such thoughts
can reveal whether the interaction between the
participants and the device is positive or negative and
can help observers to identify possible causes of errors.
Although this procedure is complex, it has the advantage
of better representing the user’s experience because,
unlike questionnaires, it is less likely to be subject to
participants’ falsifications and distortions.
Post-test phase: participants are administered
questionnaires at the end of the usability test.
Questionnaires administered in this phase, whether
validated or ad hoc built, enable observers to gather both
data about the participants’ evaluations of the device on
the whole and, if needed, about their opinion about
specific features of the device.
The procedures described so far to evaluate user
satisfaction and learn ability study subjective experience
as referred by the participants themselves, starting from a
series of standard questions. As far as learnability is
concerned, more objective procedures can be applied.
For instance, Karahoca and coll. (Karahoca et al., 2010),
asked participants to carry out tasks again twelve hours
after performing the first test. This way, researchers can
analyze the learning curve can and gather information
9. about learnability within the passing of time.
DISCUSSION AND CONCLUSIONS
In conclusion, by analysing international regulations, it
results clear that the concept of usability has acquired
more and more importance with the passing of time.
Originally, the issue of usability only concerned the field
of software and computer science, but it later spread to
the field of electro-medical technology and of medical
devices.
This paper provides a guideline to users, professionals in
usability analysis and manufacturers to carry out usability
tests based on indicators and methods which objectively
measure usability using a scientific approach both on the
level of tasks and on the level of the general device.
Efficiency and effectiveness can be assessed
quantitatively, while user satisfaction and learnability can
be estimated in a semi-quantitative way.
The new metrics can be applied to any usability
evaluation setting by taking into consideration real
experimental data deriving from usability tests and/or
estimated performance data gathered from experts'
interviews.
Regarding the context of use, usability is essential to
identify hazards and characteristics related to safety
(Figure 6) which are difficult to detect at a manufacturing
level by applying a heuristic approach only.
Even if a real medical ward represents the ideal context
of use to evaluate medical devices, the high cost and
organizational complexity, caused by the interruption of
regular activity and/or the high number of different areas
where the device is regularly used, makes the use of a
testing laboratory the best solution to combine real
environmental aspects with a controlled and multi-
configuration area (e.g. offering the possibility to carry out
simulations of worst case scenarios). Moreover, a
laboratory based approach makes it possible to take
more exact measurements (ISO 9241-11:1998).
References
Albert W, Dixon E (2003). Is this what you expected? The use
of expectation measures in usability testing. Proceedings of
the Usability Professionals Association 2003 Conference,
Scottsdale, AZ.
American Standard ANSI/AAMI HE74:2001.Human factors
design process for medical devices.
Chiu CC, Vicente KJ, Buffo-Sequeira I, Hamilton RM, McCrindle
B W (2004). Usability assessment of pacemaker
programmers. PACE, 27: 1388-1398.
Chin JP, Diel VA, Norman KL (1988). Development of an
instrument measuring user satisfaction of the human-
computer interface. Proceedings of SIGCHI
17
New York: ACM/SIGCHI. 88:213-218
Dumas J, Redish J (1999). A Practical Guide to Usability
Testing. Chicago, IL.
European Norm EN 62366:2008-01. Medical devices -
Application of usability engineering to medical devices.
European Norm EN 60601-1-6:2010-04. Medical Electrical
Equipment - Part 1: General Requirements For Basic Safety
And Essential Performance - Collateral Standard: Usability,
2011.
European Norm EN 14971. Medical devices. Application of risk
management to medical devices.
European Norm EN ISO 6385-2004. Ergonomic principles in the
design of work systems.
Follmann A, Korff A, Furtjes T, Lauer W, Kunze SC, Schmieder
K, Radermacher K (2010). Evaluation of a synergistically
controlled semiautomatic trepanation system for
neurosurgery. Conference Proceedings IEEE Engineering in
Medicine and Biology Society, 2010: 2304-2307.
Garmer K, Liljegren E, Osvalder AL, Dahlman S (2002).
Application of usability testing to the development of medical
equipment. Usability testing of a frequently used infusion
pump and a new user interface for an infusion pump
developed with a Human Factors approach. International
Journal of Industrial Ergonomics, 29: 145-159.
Hersch M, Einav S, Izbicki G (2009). Accuracy and ease of use
of a novel electronic urine output monitoring device compared
with standard manual urinometer in the intensive care unit.
Journal of Critical Care, 24: 629.e13-629.e17
International Standard ISO 9241-11:1998. Ergonomic
requirements for office work with visual display terminals
(VDTs) - Part 11: Guidance on usability.
International Standard ISO/IEC 25062:2006. Software
engineering — Software product Quality Requirements and
Evaluation (Square) — Common Industry Format (CIF) for
usability test reports.
Isomursu M, Tähti M, Väinämö S, Kuutti K (2007). Experimental
evaluation of five methods for collecting emotions in field
setting with mobile applications. International Journal of
Human-Computer Studies, 65: 404-418.
Jaspers MWM, Peute LWP, Lauteslager A, Piet JM, Bakker
PJM (2008). Pre-post evaluation of physicians’ satisfaction
with a redesigned electronic medical record system. Studies
in health technology and informatics, 136: 303-308.
Daniels J (2007). A Framework for Evaluating Usability. Journal
of Clinical Monitoring and Computing (2007) 21:323–330.
Karahoka A, Bayraktar E, Karahoka D (2010). Information
system design for a hospital emergency department: a
usability analysis of software prototypes. Journal of
Biomedical Informatics, 43: 224-232.
Kirakowski J, Porteous M, Corbett M (1992). How to use the
software usability measurement inventory: the users' view of
software quality. In: Proceedings European Conference on
Software Quality, Madrid.
Lewis JR (1991). Psychometric evaluation of an after-scenario
questionnaire for computer usability studies: the ASQ.
SIGCHI Bulletin, 23, 1: 78-81,.
Lund A (2001). Measuring usability with the USE questionnaire.
Usability and User experience Newsletter of the STC Usability
SIG.
http://www.stcsig.org/usability/newsletter/0110_measuring_wi
th_use.html
10. 18
Rubin J, Chisnell D (2008). Handbook of usability testing. How
to plan, design, and conduct effective tests. Wiley Publishing:
Indianapolis, IN.
Sauro J, Dumas JS (2009). Comparison of Three One-
Question, Post-Task Usability Questionnaires.
http://www.measuringusability.com/papers/Sauro_Dum
as_CHI2009.pdf
Shneiderman B (1987). Designing the User Interface: Strategies
for Effective Human-Computer Interaction. Addison-Wesley
Publishing Co., Reading, MA.;
Sitting DF, Kuperman GJ, Fiskio J (1999). Evaluating physician
satisfaction regarding user interactions with an electronic
medical record system. Proceeding AMIA, Annual
Symposium, 400-404.
Thayer A, Dugan TE (2009). Achieving Design Enlightenment:
Defining a New User Experience Measurement Framework.
IEEE International Professional Communication Conference,
2009.