The goal of heuristic evaluation is to identify usability problems in a system. The heuristic evaluation method employs a set of usability principles, called heuristics, to guide the evaluators in identification of usability problems. Each evaluator inspects the interface and checks the compliance of the interface with the heuristics.Heuristic evaluation is performed individually by each evaluator and then the results from different evaluators are aggregated into a set of usability problems.
During heuristic evaluation, you can freely explore an interface and identify problems or you can use scenarios to focus on a specific part of the system. In this study, we will use scenarios to limit the scope of the evaluation. You can walk through the steps of each scenario and perform them on the interface once or multiple times. Also, you are free to explore other parts of the interface to become familiar with the interface. But we are only interested in identifying problems related to the provided scenarios. For each problem that you identify, you should specify 3 components: First, the problem itself. Second, the scenario in which you identified the problem. Third, the heuristic (or heuristics) with which you found the problem. You might find problems that can’t be associated to a heuristic. For those problems, just record the problem, and scenario.
As we are going to evaluate an IT security management tool, I want to give you a quick background about this class of applications. IT security tools are components in the design, development, and maintenance of a secure information technology infrastructure. Tools like network firewalls, intrusion detection systems, and identity and access management systems are examples of IT security management tools. The environment in which these tools work has certain characteristics. First, it is complex. Everyday, new security issues arise in this environment that need to be addressed in a timely fashion. Many tasks of security administrators are not routine, and require knowledge and expertise. Second, this environment is collaborative. It means that different stakeholders in the organization need to collaborate with each other to perform their tasks. Third, the stakeholders involved in IT security management have different background. There are security administrators, managers, employees, and external contractors who need to collaborate with each other to perform IT security management tasks.
As I want to present heuristics, I will give examples about how they can be used to identify problems in an IT security management tool. The IT security management tool that I will use as a running example for this part of my presentation is a network firewall. A network firewall can be a dedicated appliance, or software running on a computer. The firewall inspects network traffic passing through it, and denies or permits passage based on a set of rules.The main functionality of a firewall is filtering the packets based on certain characteristics like the source address, destination address, port, or protocol. To do this, a firewall uses a set of rules and tries to do a certain action to packets that match a rule. For example, a rule might be “block every connection from 126.96.36.199”.At first glance, it may seem that the firewall has only one stakeholder, the security administrator who manages it. But, the tool impacts different employees in the organization. One employee might require a certain type of traffic for doing his job while others might not. Managers might want to receive reports about the performance of the firewall. Multiple security administrators might want to add rules to the firewall to address security needs of their division. In summary, using the network firewall involves complexity, collaboration, and stakeholder diversity.
Now I will present the heuristics you will be using today. For each heuristic, I will describe it and give an example of how it can be applied to an IT security management tool, in this case, a network firewall. There are 7 heuristics that I will describe. They are called ITSM heuristics. I will go through them in this tutorial and give examples of how they can be applied to IT security tools.
Assume that security admin recently blocked the access of MSN messenger to the network. If you look at this action in the context of the larger activity of managing firewall rules, the result of this action impacts other stakeholders in the organization and they should be aware about changes in the status of the activity. Therefore, the tool should be able to provide this status for the end-user. For example, when the evaluator evaluates the tool for managing rules, he should see whether the tool provides communication channels such as integrating with email to send notifications about changes in the firewall policy or publishing the list of blocked applications on the web. An important point is that when showing the status of the system, only the required information should be shown. For example showing the whole firewall policy to the user would be a breach in security.
Historical information could be in the form of use-histories by other people or the subject himself. Use histories can be employed to reflect on work and getting feedback from peers. In IT security, reflecting on work is important as the actions are performed on the system by different stakeholders. Moreover, security regulations require the system to keep a history of actions.
Assume that a new application is installed in the organization but it has trouble when making outbound connections. The security admin goes through the firewall rules and finds a rule which results in blocking connections. The security admin wants to know who added the rule to the system, when it was added, and what the reason was. If the system supports archiving of the firewall rules history, this becomes possible for the security admin.
As IT security tools have different stakeholders, tools should be able to present information in the format suitable to the target audience. Furthermore, to address complexity, providing different presentation of data allows the user to view information from different perspectives and help them understand complex scenarios. From a different view point, security admins like to combine their tools together to address different problems. Therefore, tools should be able to present their information in a way that can be used by other tools and also accept inputs from different tools.
Let’s see how it applies to our firewall example. The firewall has an internal functionality which is filtering the connections. Since the stakeholders of this firewall have different backgrounds and knowledge, the information about the blocked connections can be presented in different forms to them. For example, the security admin might be interested in details about source IP address of the blocked connections, but the manager might be interested in the total number of blocked connections in each month. Therefore, the firewall should change the representation of their internal state based on the target audience. The flexible representation of information can be also viewed from a different perspective. Since the security admins’ tasks involve addressing unknown conditions, they often combine their tools together. For example, a security practitioner used a packet analyzer tool to analyze the contents of a set of captured packets. After analyzing the content, he decides to make firewall rules for filtering such packets. If the packet analyzer can be combined with the firewall (e.g., by providing suitable output for firewall) the security admin’s work is facilitated.
As ITSM tool designers can’t predict all the conditions that a tool user might face, they should provide freedom for users to choose the way they want to perform the activities. Therefore, while there should be multiple ways for users to perform activities, constraints should be enforced by the tool so the user can only choose those paths that are not violating any of the constraints.
Suppose the organization needs to comply with a standard that mandates limiting the remote desktop access to terminals in the organization. If the firewall automatically applies the required rules based on the specific standard, it avoid mistakes by the users while it provide freedom to manage rules that are not in conflict with the standard. In the heuristic evaluation, when the evaluator evaluates the rule addition part of the firewall, he can check if the system enforces some rules based on the standards. Furthermore, the system can lay out possible actions in a hierarchical fashion for the user. For example, In the first step the user can choose what he wants to do with the rules. Based on the user’s selection, system presents different strategies for performing the task, for example adding a rule by typing it or loading the rule from a file. Again, when the user wants to add the rule using the chosen strategy, then firewall may restrict the values the user can enter based on the policy, the users position in the organization, etc.
Because the use of ITSM tools involves multiple stakeholders, the tool should provide facilities for dividing work between different stakeholders. For those tasks that have a routine procedure, incorporation of a workflow in the tool would be a good idea. But for unknown conditions, tool should provide ways for users to generate plans for performing the activity (for example, showing who is available to perform a task or allow a workflow to be created dynamically)
Suppose the firewall generates notifications when connection attempts to a certain address exceeds a threshold. If the notifications are sent to all security admins across the organization, they will have a hard time figuring out who should investigate which notification. But if the firewall helps in assigning notifications to different security admins, it will remove some of the burden.
To address problems in the complex and evolving environment of ITSM, a subject needs to use the knowledge and experience of other stakeholders involve in the activity. To facilitate accessing distributed knowledge, ITSM tools should enable their users to express their knowledge in a form of a document, web-page, or script that can be used by other users and also facilitate identification and access to the required knowledge sources for accomplishing the activity. In cases that documenting knowledge is not feasible, a method for finding and starting collaboration with the person who possesses the knowledge should be provided.
Assume that a new vulnerability has emerged over the internet. The security admins in the organization do not have the knowledge about setting up firewall rules to deal with the situation. If the firewall doesn’t provide any facility for finding the required knowledge, security admins might not be able to deal with the situation in a timely fashion. Now if the firewall allows other security admins in different organizations to share their knowledge about this vulnerability, it will address this problem. This sharing can be in form of a script, that adds required rules to the firewall, or a wiki that publishes the instructions about how the firewall should be configured to deal with the vulnerability. Knowledge sharing can be also limited to the organization itself. Recall the example of the security practitioner is trying to comprehend a rule. If the system allows sharing comments about the reason of certain rules with the rules, it will help other security admins understand the configuration.
Many actions in ITSM are responses to new, unseen and complex situations. These actions should be performed on systems that are critical to the organization. Moreover, the actions are distributed in time and space and the result of an action can't be evaluated in real time. Therefore, the cost of errors in these actions is huge. To find a solution to a new or complex problem, a security admins usually consults different information sources and combines them into a single plan (a plan, a guide document, a check list, etc.) This plan extracted from different sources might not be correct. Therefore, it should be verified before applying it to the system.Therefore, ITSM tools should allow users to rehearse the action on a non-critical, test system, evaluate the outcome of the action, and then apply it on the critical system. If something goes wrong in the rehearsal, the user can re-examine his or her interpretation of the external sources. After successful rehearsal, users can perform rehearsed actions on the critical artifact. To facilitate this process, tools should help creation of a non-critical system from a critical system, and help the process of applying rehearsed plan on the critical system.
Assume security practitioners decided to add a batch of rules to the firewall, in order to deal with certain vulnerability. Once they add the rules, some of the applications that are critical to the organization might stop working; this can result in a loss for company. Therefore, the firewall should provide facilities for security practitioners to verify their knowledge about dealing with vulnerability. They can test their knowledge on a test system before applying it on the real system. If something goes wrong while testing, security practitioners can revisit their knowledge and then test it again.
Now I will present the heuristics you will be using today. For each heuristic, I will describe it and give an example of how it can be applied to an IT security management tool, in this case, a network firewall. There are 10 heuristics that I will describe. They are called Nielsen’s heuristics and you might have seen them before. I will go through them in this tutorial to review the heuristics for you and give examples of how they can be applied to IT security tools.
The system should keep users informed about what is going on in the system, through appropriate feedback within reasonable time, so that users can accurately determine the right action to perform on the system and also can determine the result of their actions.
Assume that the security practitioner is auditing the firewall. He observes these two rules written in the firewall rule configuration file. As these rules are conflicting, the security practitioner can’t determine if the firewall allows connections to 192.168.0.1 port 22 or not. But, if the firewall provides a graphical view of its state that shows the connections that go through the firewall, the security practitioner can easily understand the status of the system.
Users can understand the system if it matches what the users know from the real world. [Nielsen heuristic itself with modifications] Therefore, the system should speak the users' language, with words, phrases, and concepts familiar to the user, rather than using system-oriented terms. It should follow real-world conventions, making information appear in a natural and logical order.
Assume there is rule in the firewall that says inbound connections on eth0 network interface should be blocked. So from the viewpoint of the security admin, inbound connections might mean connections that come from outside world into the organization’s network. But the firewall is developed in a way that inbound means connections that enter eth0, and eth0 is a network interface that connects to the internal network. Therefore inbound connections to eth0 might mean connections that come from the internal network. Here we have a mismatch between what seems logical in the real world and how the system works.
Users often make mistakes in choosing their actions. Therefore there should be a way for users to exit an undesirable state. Supporting undo or redo is an example of a shortcut for exiting such an unwanted state.
Assume the security practitioner just deleted 3 rules from the firewall by mistake. The system should provide an easy way for the user to recover those deleted rules such as providing an undo feature or providing a backup version of the rules.
When using a system, users should not have to wonder whether different words, situations, or actions mean the same thing. It is better if the system can follow platform conventions and terminology which are familiar to the user.
In the firewall, there are two ways to manage the firewall rules. The security admin can edit the rules either in a text file of the rules or through the graphical user interface of the tool. There should be a consistency in the two methods. For example, the terminology in both files should be the same. Here, in the text file the term “inbound” is used; and in the GUI the term “incoming” is used. This might lead to mistakes by security admins.
Even better than good error messages is a careful design that prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
Assume the security admin is going to add a rule that allows connections on port 22 to subnetwork 1. But there is another rule in the firewall that prohibits connections on port 22 for the whole network. In this case, the firewall can show a warning to the security practitioner, that adding such a rule might be a mistake.
Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
Assume the security admin wants to add a rule to the firewall rule set. The system can provide him with a text box to enter the rule like in this figure. But the security admin (unless he has been working with the firewall for years) needs to switch back and forth between the interface and the help file to know the syntax and the order of writing the rule.
Different users might use system in different ways. For example a novice user might prefer to go through more steps that help him do the tasks successfully, but an expert user might prefer a very straightforward way for performing an action that he is familiar with. Therefore, it would be good if the tool provides customization in a way that helps users tailor their frequent actions and choose the way they want to perform their tasks.
Assume that the security admin now gets more familiar with the firewall. He wants to easily paste certain rules into the firewall. The interface on the top, while helps him to understand and comply with syntax, does not allow him to directly paste whole rules into the system. But the second interface, on the bottom, allows this. Therefore, the system should provide the security admin the flexibly to choose what interface he wants to use. If he is comfortable with writing rules himself, the bottom interface might do the job faster.
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
Aesthetic and minimalist design could be achieved in a firewall by providing a well-designed user interface. The interface should show the information that security admins really need, no more or less. For example, in the top picture, there are no visual icons for the different components of firewall rules and there are empty rules that filled the screen. On the bottom interface, the rules are demonstrated by visual cues and only existing rules are shown.
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
Assume the security admin is going to add a rule that allows connections on port 22 to subnetwork 1. When he adds the rule, the system shows an error. As you can see in the picture, the error does not help the security practitioner understand why the rule can’t be added to the system or what he should do. Now if the system provides the reason of the error and provide suggestions for improvement, it would help the security practitioner. For example, the first and the second error messages describe the reason of the error. The third error message, gives suggestions as well.
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.
Recall the rule adding interface from the previous examples. It would be good if the system can provide a link to the help document or display a pop-up window that describes the logic behind each rule, the range of possible values, and the different parameters.
Now that we have covered the heuristics, I want to give you a quick overview on the system that you are going to evaluate. The system is an Identity Management System. I will start by describing the functionality of the system through an example.
Assume a new employee just joined neteauto, a big car insurance company, as a sales rep. The company wants her to be productive right away. So they want to give her access to the resources in the organization, but only those resources that she really requires. Neteauto uses an identity management system to manage the identities of this employee and the access privileges she needs across multiple systems. An Identity management system can automatically create identities for users on different systems and allow users to access basic services like email and active directory.
Now, assume she wants to access certain records in the sales system. She can initiate a request inside the identity management system. This request goes through a workflow and is approved by her manager (the VP of sales). After the manager’s approval, a member of the IT security team implements her request by granting her access to the records.
This system can also provide facilities for auditing. A security admin can start a certification process in the IdM system. By certification, we mean auditing the privileges of employees to check that they only have access to the resources that they need. In this process, the manager receives a notification that he should certify the privileges of 4 of his employees. So he goes through the list of roles for these 4 employees and checks if they should possess these roles or not. After the verification, those roles that are not legitimate are revoked from the users.
I will now briefly go through the interface of the tool we are going to evaluate. It is called CA Identity Manager.
Heuristic Evaluation Tutorial 1
Heuristic Evaluation• Goal: Identifying usability problems in existing systems• Checking compliance with a set of usability principles• Performed individually• Results will be aggregated 2
Heuristic Evaluation• The scope is limited to scenarios• You can go through the interface several times• You should specify: – The problem – The task – The heuristic(s) 3
IT security management (ITSM) tools• IT security management tools are components in the design, development, and maintenance of a secure information technology infrastructure. – Examples: network firewall, intrusion detection system, identity and access management system• Characteristics of the ITSM environment – Complex, collaborative, people with different backgrounds 4
Example ITSM Tool• Firewall as a running example• Filtering packet based on certain characteristics• Firewall use rules for this purpose – E.g. block every connection from 188.8.131.52 5
1 – Visibility of activity statusProvide users with awareness about the status of the activity distributed over time and space, including the other users involved in the activity, their actions, and distribution of work between them; rules that govern the activity; tools, information, and material that are used in the activity; and progress toward the activity objective. Provide communication channels for transferring the status of the activity. While providing awareness is crucial, provide awareness only about what a user needs to know to complete his actions. 7
1 – Visibility of activity status Firewall Security admin Providing communication channels Provide shared view of the system state Provide information about who is responsible Don’t show all the firewall rulesEmployee 8
2- History of actions and changes on artifactsAllow capturing the history of actions and changes on tools or other artefacts such as policies, logs, and communication between users. Provide a means for searching and analyzing historical information. 9
2- History of actions and changes on artifacts FirewallSecurity Admin 1 - Provide archiving - History of actions - Data correlation and filtering 10Security admin 2
3- Flexible representation of informationAllow changing the representation of information to suit the target audience and their current task. Support flexible reports. Allow tools to change the representation of their input/output for flexible combination with other tools. 11
3- Flexible representation of information What is the source ip addresses?Security Admin Total number/month? Manager Malicious network activity Firewall Packet analyzer • Different presentation formats or multiple views • Flexible reporting • Different methods of interaction with the tool • Combinable tools • Customizable tools 12
4- Rules and constraintsPromote rules and constraints on ITSM activities, butprovide freedom for users to choose different pathsthat respect the constraints. Constraints can beenforced in multiple layers. For example, a tool couldconstrain the possible actions based on the task, thechosen strategy for performing the task (e.g., the orderof performing actions), the social and organizationalstructure (e.g., number of subjects involved in thetask, policies, standards), and the competency of theuser. 13
4- Rules and constraints Standards? Organization policy? … … …Security Admin Firewall - Allow application of different policies - List actions in hierarchical fashion 14
5- Planning and dividing work between usersFacilitate dividing work between the users involved in an activity. For routine and pre-determined tasks, allow incorporation of a workflow. For unknown conditions, allow generation of new work plans and incorporation of new users. 15
5- Planning and dividing work between users Subdomain A notifications Security Admin 1 Subdomain B notifications Security Admin 2 • Support workflow Subdomain C notifications • Task prioritization Security Admin 3 16
6- Capturing, sharing, and discovery of knowledgeAllow users to capture and store their knowledgeexplicitly by generating documents, web-pages, scripts, and notes or implicitly by providingaccess to a history of their previous actions. Tools couldthen facilitate sharing such knowledge with otherusers. Furthermore, tools should facilitate discovery ofthe required knowledge source including artefacts or aperson who possess the knowledge and provide meansof communicating with the person who possesses theknowledge. 17
6- Capturing, sharing, and discovery of knowledge Firewall in organization 1 There is a new attack that exploits port 22 …. Security Admin in Organization 2 internet- Knowledge sharing Security admin in- Support scripts organization 1- Tagging / Wiki / Social Navigation- Communication channel Firewall in organization 2 18
7- Verification of knowledgeFor critical ITSM activities, tools should help SPs validate their knowledge about the actions that are required to perform the activity. Allow users to validate their knowledge by performing actions and validating the results on a test system before applying them to the real system. Allow users to document the required actions in the form of a note or a script; this helps the users or their colleagues to review the required actions before applying them on the system. 19
7- Verification of knowledgeOnline sources in the internet Test Security admin Real • Rehearsal and planning • Manageable configuration 20
1 - Visibility of system statusThe system should always keep users informed about what is going on, through appropriate feedback within reasonable time. 22
1 - Visibility of system status : example Firewall rules: Block all connections to the range(192.168.0.1 - 192.168.0.255) Allow connections to 192.168.0.1 port 22Security Admin Firewall 23
2 - Match between system and the real worldThe system should speak the users language,with words, phrases and concepts familiar to theuser, rather than system-oriented terms. Itshould follow real-world conventions, makinginformation appear in a natural and logicalorder. 24
2 - Match between system and the real world : example Firewall rules: eth0 inbound on port 22 block eth 1 eth 0 Security Admin 25
3 - User control and freedomUsers often choose system functions by mistakeand will need a clearly marked "emergency exit"to leave the unwanted state without having togo through an extended dialogue. Support undoand redo. 26
3 - User control and freedom : example Firewall rules: A X B X C X Firewall 27
4 - Consistency and standardsUsers should not have to wonder whetherdifferent words, situations, or actions mean thesame thing. Follow platform conventions andterminology which are familiar to the user. 28
4 - Consistency and standards : example Firewall rules file: eth0 inbound … block Firewall Security Admin Firewall UI: Block all incoming connections on eth 0 Block all incoming connections on eth 1 Block all incoming connections on eth 2 … 29
5 - Error preventionEven better than good error messages is acareful design which prevents a problem fromoccurring in the first place. Either eliminateerror-prone conditions or check for them andpresent users with a confirmation option beforethey commit to the action. 30
5 - Error prevention : example Firewall rules: Block all connections on port 22 Are you sure … ?Security Admin Allow connections on port 22 for subnetwork1 31
6 - Recognition rather than recallMinimize the users memory load by makingobjects, actions, and options visible. The usershould not have to remember information fromone part of the dialogue to another. Instructionsfor use of the system should be visible or easilyretrievable whenever appropriate. 32
6 - Recognition rather than recall : exampleSecurity Admin Firewall Write thethe rulethethe following: box: Write rule in in following text interface From To Port 33
7 - Flexibility and efficiency of useAccelerators -- unseen by the novice user -- mayoften speed up the interaction for the expertuser such that the system can cater to bothinexperienced and experienced users. Allowusers to tailor frequent actions. 34
7 - Flexibility and efficiency of use : example Write the rule in the following: interface From To PortSecurity Admin Write the rule in the following text box: 35
8 - Aesthetic and minimalist designDialogues should not contain information whichis irrelevant or rarely needed. Every extra unit ofinformation in a dialogue competes with therelevant units of information and diminishestheir relative visibility. 36
9 - Help users recognize, diagnose, and recover from errorsError messages should be expressed in plainlanguage (no codes), precisely indicate theproblem, and constructively suggest a solution. 38
9 - Help users recognize, diagnose, and recover from errors : example • Error – Rule is in conflict with the rule number 34 • Error – The port number is not valid • Error – The port number should between 1~1024 • Error • Error 22 • Error – rule can’t be added Security Admin Firewall Allow connections on port 22 for subnetwork1 39
10 - Help and documentationEven though it is better if the system can beused without documentation, it may benecessary to provide help and documentation.Any such information should be easy to search,focused on the users task, list concrete steps tobe carried out, and not be too large. 40
10 - Help and documentation : exampleSecurity admin Firewall Write the rule in the following text box: Write the rule in the following: interface From To Port ? Firewall UI 41
The target system for heuristic evaluationIDENTITY MANAGEMENT SYSTEM 42