The goal of heuristic evaluation is to identify usability problems in a system. The heuristic evaluation method employs a set of usability principles, called heuristics, to guide the evaluators in identification of usability problems. Each evaluator inspects the interface and checks the compliance of the interface with the heuristics. Heuristic evaluation is performed individually by each evaluator and then the results from different evaluators are aggregated into a set of usability problems.
During heuristic evaluation, you can freely explore an interface and identify problems or you can use scenarios to focus on a specific part of the system. In this study, we will use scenarios to limit the scope of the evaluation. You can walk through the steps of each scenario and perform them on the interface once or multiple times. Also, you are free to explore other parts of the interface to become familiar with the interface. But we are only interested in identifying problems related to the provided scenarios. For each problem that you identify, you should specify 3 components: First, the problem itself. Second, the scenario in which you identified the problem. Third, the heuristic (or heuristics) with which you found the problem. You might find problems that can’t be associated to a heuristic. For those problems, just record the problem, and scenario.
As we are going to evaluate an IT security management tool, I want to give you a quick background about this class of applications. IT security tools are components in the design, development, and maintenance of a secure information technology infrastructure. Tools like network firewalls, intrusion detection systems, and identity and access management systems are examples of IT security management tools. The environment in which these tools work has certain characteristics. First, it is complex. Everyday, new security issues arise in this environment that need to be addressed in a timely fashion. Many tasks of security administrators are not routine, and require knowledge and expertise. Second, this environment is collaborative. It means that different stakeholders in the organization need to collaborate with each other to perform their tasks. Third, the stakeholders involved in IT security management have different background. There are security administrators, managers, employees, and external contractors who need to collaborate with each other to perform IT security management tasks.
As I want to present heuristics, I will give examples about how they can be used to identify problems in an IT security management tool. The IT security management tool that I will use as a running example for this part of my presentation is a network firewall. A network firewall can be a dedicated appliance, or software running on a computer. The firewall inspects network traffic passing through it, and denies or permits passage based on a set of rules. The main functionality of a firewall is filtering the packets based on certain characteristics like the source address, destination address, port, or protocol. To do this, a firewall uses a set of rules and tries to do a certain action to packets that match a rule. For example, a rule might be “block every connection from 188.8.131.52”. At first glance, it may seem that the firewall has only one stakeholder, the security administrator who manages it. But, the tool impacts different employees in the organization. One employee might require a certain type of traffic for doing his job while others might not. Managers might want to receive reports about the performance of the firewall. Multiple security administrators might want to add rules to the firewall to address security needs of their division. In summary, using the network firewall involves complexity, collaboration, and stakeholder diversity.
Now I will present the heuristics you will be using today. For each heuristic, I will describe it and give an example of how it can be applied to an IT security management tool, in this case, a network firewall. There are 7 heuristics that I will describe. They are called ITSM heuristics. I will go through them in this tutorial and give examples of how they can be applied to IT security tools.
Assume that security admin recently blocked the access of MSN messenger to the network. If you look at this action in the context of the larger activity of managing firewall rules, the result of this action impacts other stakeholders in the organization and they should be aware about changes in the status of the activity. Therefore, the tool should be able to provide this status for the end-user. For example, when the evaluator evaluates the tool for managing rules, he should see whether the tool provides communication channels such as integrating with email to send notifications about changes in the firewall policy or publishing the list of blocked applications on the web. An important point is that when showing the status of the system, only the required information should be shown. For example showing the whole firewall policy to the user would be a breach in security.
Historical information could be in the form of use-histories by other people or the subject himself. Use histories can be employed to reflect on work and getting feedback from peers. In IT security, reflecting on work is important as the actions are performed on the system by different stakeholders. Moreover, security regulations require the system to keep a history of actions.
Assume that a new application is installed in the organization but it has trouble when making outbound connections. The security admin goes through the firewall rules and finds a rule which results in blocking connections. The security admin wants to know who added the rule to the system, when it was added, and what the reason was. If the system supports archiving of the firewall rules history, this becomes possible for the security admin.
As IT security tools have different stakeholders, tools should be able to present information in the format suitable to the target audience. Furthermore, to address complexity, providing different presentation of data allows the user to view information from different perspectives and help them understand complex scenarios. From a different view point, security admins like to combine their tools together to address different problems. Therefore, tools should be able to present their information in a way that can be used by other tools and also accept inputs from different tools.
Let’s see how it applies to our firewall example. The firewall has an internal functionality which is filtering the connections. Since the stakeholders of this firewall have different backgrounds and knowledge, the information about the blocked connections can be presented in different forms to them. For example, the security admin might be interested in details about source IP address of the blocked connections, but the manager might be interested in the total number of blocked connections in each month. Therefore, the firewall should change the representation of their internal state based on the target audience. The flexible representation of information can be also viewed from a different perspective. Since the security admins’ tasks involve addressing unknown conditions, they often combine their tools together. For example, a security practitioner used a packet analyzer tool to analyze the contents of a set of captured packets. After analyzing the content, he decides to make firewall rules for filtering such packets. If the packet analyzer can be combined with the firewall (e.g., by providing suitable output for firewall) the security admin’s work is facilitated.
As ITSM tool designers can’t predict all the conditions that a tool user might face, they should provide freedom for users to choose the way they want to perform the activities. Therefore, while there should be multiple ways for users to perform activities, constraints should be enforced by the tool so the user can only choose those paths that are not violating any of the constraints.
Suppose the organization needs to comply with a standard that mandates limiting the remote desktop access to terminals in the organization. If the firewall automatically applies the required rules based on the specific standard, it avoid mistakes by the users while it provide freedom to manage rules that are not in conflict with the standard. In the heuristic evaluation, when the evaluator evaluates the rule addition part of the firewall, he can check if the system enforces some rules based on the standards. Furthermore, the system can lay out possible actions in a hierarchical fashion for the user. For example, In the first step the user can choose what he wants to do with the rules. Based on the user’s selection, system presents different strategies for performing the task, for example adding a rule by typing it or loading the rule from a file. Again, when the user wants to add the rule using the chosen strategy, then firewall may restrict the values the user can enter based on the policy, the users position in the organization, etc.
Because the use of ITSM tools involves multiple stakeholders, the tool should provide facilities for dividing work between different stakeholders. For those tasks that have a routine procedure, incorporation of a workflow in the tool would be a good idea. But for unknown conditions, tool should provide ways for users to generate plans for performing the activity (for example, showing who is available to perform a task or allow a workflow to be created dynamically)
Suppose the firewall generates notifications when connection attempts to a certain address exceeds a threshold. If the notifications are sent to all security admins across the organization, they will have a hard time figuring out who should investigate which notification. But if the firewall helps in assigning notifications to different security admins, it will remove some of the burden.
To address problems in the complex and evolving environment of ITSM, a subject needs to use the knowledge and experience of other stakeholders involve in the activity. To facilitate accessing distributed knowledge, ITSM tools should enable their users to express their knowledge in a form of a document, web-page, or script that can be used by other users and also facilitate identification and access to the required knowledge sources for accomplishing the activity. In cases that documenting knowledge is not feasible, a method for finding and starting collaboration with the person who possesses the knowledge should be provided.
Assume that a new vulnerability has emerged over the internet. The security admins in the organization do not have the knowledge about setting up firewall rules to deal with the situation. If the firewall doesn’t provide any facility for finding the required knowledge, security admins might not be able to deal with the situation in a timely fashion. Now if the firewall allows other security admins in different organizations to share their knowledge about this vulnerability, it will address this problem. This sharing can be in form of a script, that adds required rules to the firewall, or a wiki that publishes the instructions about how the firewall should be configured to deal with the vulnerability. Knowledge sharing can be also limited to the organization itself. Recall the example of the security practitioner is trying to comprehend a rule. If the system allows sharing comments about the reason of certain rules with the rules, it will help other security admins understand the configuration.
Many actions in ITSM are responses to new, unseen and complex situations. These actions should be performed on systems that are critical to the organization. Moreover, the actions are distributed in time and space and the result of an action can't be evaluated in real time. Therefore, the cost of errors in these actions is huge. To find a solution to a new or complex problem, a security admins usually consults different information sources and combines them into a single plan (a plan, a guide document, a check list, etc.) This plan extracted from different sources might not be correct. Therefore, it should be verified before applying it to the system. Therefore, ITSM tools should allow users to rehearse the action on a non-critical, test system, evaluate the outcome of the action, and then apply it on the critical system. If something goes wrong in the rehearsal, the user can re-examine his or her interpretation of the external sources. After successful rehearsal, users can perform rehearsed actions on the critical artifact. To facilitate this process, tools should help creation of a non-critical system from a critical system, and help the process of applying rehearsed plan on the critical system.
Assume security practitioners decided to add a batch of rules to the firewall, in order to deal with certain vulnerability. Once they add the rules, some of the applications that are critical to the organization might stop working; this can result in a loss for company. Therefore, the firewall should provide facilities for security practitioners to verify their knowledge about dealing with vulnerability. They can test their knowledge on a test system before applying it on the real system. If something goes wrong while testing, security practitioners can revisit their knowledge and then test it again.
Now that we have covered the heuristics, I want to give you a quick overview on the system that you are going to evaluate. The system is an Identity Management System. I will start by describing the functionality of the system through an example.
Assume a new employee just joined neteauto, a big car insurance company, as a sales rep. The company wants her to be productive right away. So they want to give her access to the resources in the organization, but only those resources that she really requires. Neteauto uses an identity management system to manage the identities of this employee and the access privileges she needs across multiple systems. An Identity management system can automatically create identities for users on different systems and allow users to access basic services like email and active directory.
Now, assume she wants to access certain records in the sales system. She can initiate a request inside the identity management system. This request goes through a workflow and is approved by her manager (the VP of sales). After the manager’s approval, a member of the IT security team implements her request by granting her access to the records.
This system can also provide facilities for auditing. A security admin can start a certification process in the IdM system. By certification, we mean auditing the privileges of employees to check that they only have access to the resources that they need. In this process, the manager receives a notification that he should certify the privileges of 4 of his employees. So he goes through the list of roles for these 4 employees and checks if they should possess these roles or not. After the verification, those roles that are not legitimate are revoked from the users.
I will now briefly go through the interface of the tool we are going to evaluate. It is called CA Identity Manager.
Provide users with awareness about the status of the activity distributed over time and space, including the other users involved in the activity, their actions, and distribution of work between them; rules that govern the activity; tools, information, and material that are used in the activity; and progress toward the activity objective. Provide communication channels for transferring the status of the activity. While providing awareness is crucial, provide awareness only about what a user needs to know to complete his actions.
1 – Visibility of activity status Providing communication channels Provide shared view of the system state Provide information about who is responsible Don’t show all the firewall rules Security admin Employee Firewall
Allow capturing the history of actions and changes on tools or other artefacts such as policies, logs, and communication between users. Provide a means for searching and analyzing historical information.
2- History of actions and changes on artifacts
Allow changing the representation of information to suit the target audience and their current task. Support flexible reports. Allow tools to change the representation of their input/output for flexible combination with other tools.
3- Flexible representation of information Malicious network activity What is the source ip addresses? Total number/month? Packet analyzer
Promote rules and constraints on ITSM activities, but provide freedom for users to choose different paths that respect the constraints. Constraints can be enforced in multiple layers. For example, a tool could constrain the possible actions based on the task, the chosen strategy for performing the task (e.g., the order of performing actions), the social and organizational structure (e.g., number of subjects involved in the task, policies, standards), and the competency of the user.
Facilitate dividing work between the users involved in an activity. For routine and pre-determined tasks, allow incorporation of a workflow. For unknown conditions, allow generation of new work plans and incorporation of new users.
5- Planning and dividing work between users Subdomain A notifications Subdomain B notifications Subdomain C notifications
6- Capturing, sharing, and discovery of knowledge
Allow users to capture and store their knowledge explicitly by generating documents, web-pages, scripts, and notes or implicitly by providing access to a history of their previous actions. Tools could then facilitate sharing such knowledge with other users. Furthermore, tools should facilitate discovery of the required knowledge source including artefacts or a person who possess the knowledge and provide means of communicating with the person who possesses the knowledge.
6- Capturing, sharing, and discovery of knowledge There is a new attack that exploits port 22 ….
Tagging / Wiki / Social Navigation
Security Admin in Organization 2 Security admin in organization 1 Firewall in organization 1 Firewall in organization 2 internet
For critical ITSM activities, tools should help SPs validate their knowledge about the actions that are required to perform the activity. Allow users to validate their knowledge by performing actions and validating the results on a test system before applying them to the real system. Allow users to document the required actions in the form of a note or a script; this helps the users or their colleagues to review the required actions before applying them on the system.