Adam Wilson AP Final Project REVISED FINAL DRAFT

A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 1
A Structured Approach to Systems Projects
Using Design Checklists
By Adam Wilson
DePaul University
Author Note
This paper was prepared for the Advanced Project course taught by Ed Paulson, Winter Quarter 2015.

Introduction
This paper addresses the problem of success in designing information systems. The best system
design occurs when a designer is familiar with good design approaches and can apply them appropriately
given the particular situation. Outcomes are less successful when designers do not choose intentionally
or skillfully but instead use approaches that are familiar or comfortable for them (Saffer, 2010). The
solution to this problem I will explore in this paper is the use of design checklists. The checklists will
consist of critical and easy to miss tasks taken from the Systems Development Lifecycle methodology as
well as other scholarly sources and my own personal experience. I hope to show in this paper that
checklists can help professionals in real situations to “make sense of… complex issues… in decision making
…” (McKeown, 2013). I will also address a number of obstacles to the successful use of checklists.
Obstacles include production pressures, personal inclinations, overloaded or unclear checklist design and
imposition of overly restrictive requirements (Katherine Radeka, 2015). There are three outcomes I hope
to achieve through my efforts. I would like my checklists to get used in actual design work, I would like to
achieve more consistent success in creating information systems and I would like to use the checklists in
project evaluations to better understand the influence of particular design tasks on system outcomes.
Advanced Project Competency Statements
F-11: Can design and produce a significant product that gives evidence of advanced competence.
F-12: Can design practical, holistic design checklists that allow flexibility while controlling for coverage of
primary value adding, critical and easy to miss project tasks.
1. Can describe primary value adding, critical and easy to miss project tasks in computer systems design.
2. Can define strategic pause points for checklist review amidst production pressures.
3. Can describe how checklists help individuals or teams take complex knowledge and consistently, correctly
and safely apply it under pressures and constraints.
The Problem
In today’s knowledge workforce end-users “design and develop an increasing number of their
own applications” (Turban, 2009). The discipline of design is therefore of increasingly broad relevance.
The best design occurs when the designer can apply the best approaches for the particular situation
however many designers include only those with which they are familiar or most comfortable (Saffer,
2010). The many phases and tasks of design, often with sequential dependencies present a problem of
complexity. These challenges can result in delays, abandoned projects, lost resources, communication
failures and tensions between stakeholders, poor usability, unmet system requirements, reduced internal
and external efficiency and effectiveness, poor usage rates and lost organizational knowledge (Plessis,

2007) and the dissolution of relationships. Even when we have years of experience and training,
“avoidable failures continue to plague us… the volume and complexity of knowledge… [Can] exceed our
ability as individuals to properly deliver it…” (Gawande, 2009). I’ve had varied success with system
projects, facing many of the challenges described above not always understanding the design tasks that
might help or how to consistently integrate them under constraints of time, budget, expertise, etc..
My Background
From 1999 to 2009 I worked as an office manager in a small private school. I oversaw customer
service and developed workflows for staff-client interactions. In 2009 I left this position and returned to
school at DePaul to complete my BA in Computer Information Systems. I hoped to bring my interest in
administrative process to the digital space as administrative work was becoming increasingly digital. The
Information Systems Focus Area courses have covered technical aspects of computing such as
programming and administering systems as well as people and process oriented aspects like systems
analysis and design, product and project management and organizational dynamics. I was in school for 2
years before I took another full time job as office manager at a church I admired. The church hoped that I
could bring some of their office processes “into the digital age”. Over that time I partnered with various
staff and others to design, built and support new systems for a number of core processes. These projects
included a public website redesign, implementation of Google docs and sites for real time document co-
authoring, an online room reservation system that has handled thousands of transactions and an online
staff resource portal among others.
Solution Overview: The Design Checklist
My solution to the problem of achieving consistent system design success is a design checklist.
Research reveals that checklists are powerful but they need to be created and implemented well
(Katherine Radeka, 2015). Checklists are everywhere in various forms from aircraft flight checklists to
construction site work schedules to rubrics used by teachers to grade assignments. Checklists are used to
support work by individuals and teams in complex, fast paced environments full of production pressures,
cultural factors, personal inclinations and regulatory issues (Asaf Degani, 1991). They can help
professionals “make sense of… complex issues… in decision making situations…” (McKeown, 2013).
Important checklist best practices I will incorporate include customization to individual needs, inclusion of
only critical or easy to miss items so the lists are non-cumbersome, a series of short lists instead of one
long list and usage guidelines for when and by whom each list should be reviewed (Katherine Radeka,
2015). The lists will be populated by design tasks taken from the Software Development Lifecycle (SDLC)
(Turban, 2009) and other methodologies including Interaction Design and User-Centered Design (UCD)
among others.

Intended Outcomes: Success, Consistency, Understanding
My intended outcome from the design checklists is consistent system design success. I would
also like to better understand the linkages between design tasks and outcomes and be more intentional
in choosing how to spend my time and energy. The next section on evaluation will discuss ways of
measuring success. I would like to replicate the outcomes I’ve achieved using checklists for my tasks as an
office manager. The partial image of my office management weekly checklist below demonstrates that
my prior work in administration has been highly checklist driven. Although I had not yet studied checklist
best practices the lists I created for various aspects of my job follow some of them. They include easy to
miss and critical items, fit on one page, are quickly reviewable and ordered sequentially and/or logically,
were used collaboratively by administrative staff to track and communicate among team members and
were online to facilitate sharing and frequent updates (Katherine Radeka, 2015). Comprehensive
instructions were hyperlinked to items to facilitate access without overloading the checklist itself. The
lists were built in Google Sheets and the Design Checklists at the end of this paper may eventually be
migrated there as well. My office management checklists helped make me a very reliable and consistent
administrator and I would like to expand that kind of competency into the slightly less routinized and
more creative space of design.
Office management checklist used by the author. Checklists helped make me a very reliable and consistent
administrator and I’d like to generalize this to systems design.
Criteria and Methods of Checklist Evaluation
In the next few pages I’ll describe some criteria that makeup successful system outcomes as well
as how to evaluate them and at what stages of design projects. The activities will become part of my
design checklists. Joel Palmius states in the article Criteria for measuring and comparing information
systems (Palmius, 2007) that “With operational criteria… a foundation can be laid for improving the
system… A designer needs to know what the goal is, and a buyer… needs to know whether the goals have
been fulfilled. With criteria, there could be a checklist before, during and after a design phase… (Palmius,
2007).” Evaluations are conducted by designers looking independently at analytics, transactions and

other data but the sections below focus mostly on usability factors and subjective stakeholder feedback
via questionnaires and observation.
Evaluating the Existing System (Pre-Design)
The first half of the items on a Checklist for User-Centered Design (Oregon State Computer
Engineering Dept., 2000) address “Initial motivation” and “pre-Design Activities”. These activities involve
user’s right from the start with evaluation of the current system and other existing competitive systems.
This is crucial for establishing baseline measures against which to compare any new systems designed. It
is also crucial to put users at the center of the design process from the outset. User concerns involve
work goals, tasks, mental models, needs and desires. This pre-design evaluation also investigates
whether the system is actually the main problem as opposed to factors like user awareness, training or
trust in the current system.
Multiple Evaluations during the Development Lifecycle
Evaluation should occur at various points during design. Methods that involve multiple iterations
of a solution in the design, prototype and eventually the final system need to evaluate repeatedly.
Various methodologies all count on feedback at points (Turban, 2009). Examples include pre-design,
formal design proposal (design brief/system requirements document), low-fidelity prototypes like
wireframes, initial system rollouts, and ongoing during the operation of the system.
Project Management and Collaborative Criteria for Success
Communication is a prerequisite for design success. In the building construction industry experts
have said that communication and tracking advances are the biggest improvements to the trade in recent
decades (Gawande, 2009). Consultant Scott Vaughn (Vaughan) goes so far as to recommend signed
agreements between parties in settings that are usually more informal such as between colleagues. In
today’s workplace where “rapid expert design” is going on all the time (Saffer, 2010) the “designer” is
often a member of the team. The table below lists some metrics and ways to control for them.
Project management success Criteria Project Management Tasks (Coleman, 2015)
 Project completion rates
 Project delays
 Failures of communication
 Stakeholder relationships and
morale
 upfront discussion of budget, timeline and goals
 client questionnaire
 defined stages and deliverables
 designer proposal including features, costs, stages
and deliverables
 contract to sign with paid deposit
Use, Validation and Refinement of the Checklists

We cannot evaluate the impact of the checklist on design projects unless we are using the
checklist. Project conditions present challenges for the successful implementation of checklists (Asaf
Degani, 1991). Checklists deal with complex situations as we’ll explore in the upcoming literature later in
this paper. Nobody gets them right on the first try and so they must be constantly reviewed, validated
and updated.
Designer Bias and Evaluation: Who Should Conduct Evaluations
Neutrality and freedom from bias is important in testing. The designer should not conduct or
collect evaluations if possible. If the designer must conduct evaluations they may choose not to identify
themselves as the designer to mitigate censorship (Saffer, 2010) and politeness by overtly inviting critique
and improvement, making clear that the design is a hypothesis. Usability testing specialists are often
involved in larger projects to navigate these issues and bring findings back to the design team.
Recruiting Subjects
The same techniques used for recruiting subjects for design input can be used for design
evaluation. In a website overhaul I performed I recruited ten users for design input. It’s a good practice
to thank people for input and to update them on what happened with the project (Saffer, 2010). I did
thank them but did not recruit them to evaluate the final product. I could have done so. New subjects can
also be recruited.
Observation and Interviews
Observation and interviews are two primary forms of evaluation that can work together. Design
rules apply when using observation: Go to the users (or stakeholders), use their computer, sit at their
desks, talk to them, take notes (Saffer, 2010). Bucket testing can be done where users are shown two
different design options and feedback is used to rank them (Saffer, 2010). A Test Plan should be devised
containing neutral questions as well as test routes “through the product that test functionality” (Saffer,
2010). Cognitive and behavioral measures like “time taken to complete a task or the number of errors or
deviations from a critical path” can be recorded (Jefsioutine, 2006). Dan Saffer adds that stakeholder or
user interviews are done best individually and in person to reveal thinking, emotion, as an opportunity to
challenge and should be conducted with stakeholders across all levels on the organizational chart especially
including those with power and/or direct client contact (Saffer, 2010). All observations and patterns should be
recorded and used to produce a summative “Opportunity Report” showing trouble spots and suggestions
for improvement (Saffer, 2010). Saffer suggests specific questions for Stakeholder interviews which
appear in the appendix of this paper.
Quick and Dirty Usability Evaluation using Heuristics
Prominent Designer Dan Saffer says that “if you don’t have the resources to do testing with users,
the least you can do is perform a heuristic evaluation” (Saffer, 2010). Heuristic evaluation is based on the

ten Usability Heuristics developed by Jacob Nielsen in 1990 and refined based on factor analysis of 249
usability problems (Nielsen, 1995). These heuristics were the basis for the rapid system usability
improvements discussed in the source elsewhere cited in this project at the University of Bari (Nicola
Convertini, 2004) and they are listed in an appendix to this document. They are broad rules of thumb for
what constitutes a useable system and include items such as “visibility of system status”, “consistency
and standards” etc. (Nielsen, 1995).
Quick and Dirty Evaluation using The System Usability Scale (SUS) Questionnaire
The SUS has been widely used and “provides a “quick and dirty”, reliable tool for measuring
usability (usability.gov, 2015). It consists of a 10 item questionnaire with five response options for
respondents: from strongly agree to strongly disagree.” It has become an industry standard with
references in over 1300 articles and publications. Notable benefits of the SUS include easy administration,
usability “on small sample sizes with reliable results” and proven validity for differentiating usable and
unusable systems. It does not diagnose particular issues but is useful for classifying overall usability
(usability.gov, 2015). SUS original creator John Brooke defends the concise nature of the questionnaire
saying that it is often “desirable to have measures which do not require vast effort and expense to collect
and analyze”, claiming that questionnaires over 25 questions are often not completed (Brooke, 1986).
Brooke goes on to state that systems are like apples and oranges and that the context of system use
varies widely ranging from word processing to chemical manufacturing. He argues therefore that a
subjective assessment measure like the SUS is the kind of evaluation tool which remains appropriate
across different system. The SUS is ideally administered just after the tester has used system and before
further discussion about the experience (Brooke, 1986). The SUS appears in the Appendix to this paper.
Questionnaire Items Supplementary to the SUS
The following questions are not directly addressed by the SUS. They could be part of a separate
evaluation questionnaire with follow-up interactions. User and stakeholder input is so important to good
design that I’ve taken the time to gather questions from additional survey sources and a few from my
own experiences that are not addressed directly by the SUS. I have limited them to 25 per the
recommendations by SUS article author John Brooke (Brooke, 1986). Like with the SUS the questions are
strong statements designed to minimize answers in the middle range of a 1-5 strongly disagree to
strongly agree Likert scale. The questionnaire is included in the Appendix of the paper.
Literature Review: Checklist Design and Human Factors
Atul Gawande writes in The Checklist Manifesto (Gawande, 2009) that “the volume and
complexity of knowledge today has exceeded our ability as individuals to properly deliver it to people –
consistently, correctly, safely. We can do better, using the simplest of methods: the checklist.” Research

reveals that checklists are indeed, very powerful but they need to be created and implemented well. A
checklist is not so simple after all. Checklists are everywhere in various forms from aircraft flight
checklists to construction site work schedules and rubrics used by teachers to grade assignments.
Checklists are used to support work by individual and teams or even teams of teams in complex, fast
paced environments full of production pressures, cultural factors, personal inclinations and regulatory
issues (Asaf Degani, 1991). These conditions and present challenges for the design and successful
implementation of checklists.
Buy-In: Making Checklist Usage a Norm
Buy-In is a critical factor in creating and implementing checklists. There are various strategies for
making checklist usage a norm including co-creation among professionals or in client-professional
relationships (McKeown, 2013), requiring users to make their own checklists and share them as part of
work interactions (Katherine Radeka, Whittier Consulting Group, Inc., 2015), ethnographic observations
of checklist usage behaviors and usage context, consideration of human end-user and cultural factors, list
usability and phraseology (Asaf Degani, 1991) among others. My wife is a teacher and she shared with
me the ways in which she is required to submit lesson plans. She is free to create or retrieve the plans
from a source or colleague but they must meet particular criteria. This is similar to the ideas put forth by
the “Experience Design Framework” (EDF) which is a response to methods that are stricter. The authors
argue that frameworks such as the EDF can work to create a “principled context without being overly
prescriptive” (Jefsioutine, 2006).
Top Leadership Support: power, norms, expertise
Checklists are becoming more common in medicine as studies reveal dramatic impacts on rate of
infection and other outcomes (Gawande, 2009). “Americans… have upwards of 150,000 deaths following
surgery every year – more than three times the number of road traffic fatalities. Moreover research has
shown that at least half… are avoidable.” IN Michigan, a project which rolled out a checklist to prevent
central line infections and in 2006 the New England Journal of medicine published findings revealing a
66% drop in these infections statewide (Gawande, 2009). Similar dramatic gains are being seen
worldwide with more comprehensive surgical checklists being implemented. Despite dramatic evidence
like this, Professional norms and operations issues can remain major challenges in the implementation of
a checklist. Programs must be holistic and involve dedicated project managers with expertise as well as
top level leadership. For example, nurses may need to be empowered to break the norm of deferring to
surgeons and be required with absolute backing from top hospital leadership to point out when a surgeon
has omitted a step on the checklist. Top executives may need to personally conduct on-site observations
which may reveal issues like failure of operations staff to consistently provide vital surgical supplies to
operating rooms. Only top leadership can address these kinds of holistic system issues. Surgical kits are

being redesigned by manufacturers to include all necessary supplies packaged together and ready and
props are being introduced to provide forcing functions on behavior such as scalpel cases inscribed with
phrases like “ready for takeoff?”, a reference to the aircraft industry which pioneered the use of
checklists to deal with complexity (Gawande, 2009).
List Design
I’ve discussed issues in implementation first above because I have personally tended to focus
more on the design of the checklist product and less on the context of use. There is a wealth of crucial
research on the design of the lists themselves and there is overlap between these two areas. List design
should be human-centered and user-friendly for example, in order to increase utilization (Asaf Degani,
1991).
Key terminology is discussed in list design requirements. Lists should be grounded in the
literature (McKeown, 2013), in many cases non-comprehensive containing only the easy to miss or “killer”
(critical) items and no longer than a single letter sized page, (Katherine Radeka, Whittier Consulting
Group, Inc., 2015) a guide to use should be included and the formatting should be logical incorporating
location and most crucially, timing of activities so they are not competing with other production activities
directly and the list can be completed consistently at a specific break point in workflow (Asaf Degani,
1991). The World Health Organization Surgical Safety Checklist pictured below (World Health
Organization, 2009) exemplifies some of these principles: Sections are titled by when they should occur, a
list of required participants in the check are specified, the whole thing fits on a single sheet of paper and
the list has been edited down through validation and testing to only the most critical or easy to miss
items.

Impact of the Literature on Design Checklist Decision Making
My research has pointed to some best practices I will try to incorporate in my design checklist as
follows below.
 Design phases will each have their own list according to logical pause points.
 There will be no more than ten items per list.
 The deliverable for each phase will be a checklist item.
 Activities will be drawn from widely accepted best practices.
 Appropriate list review participants will be stated on the lists.
 Each item will have a large checkbox to mark appropriately.
 Each item will represent a critical or easy to miss item. Optional additional or specific activities
may be listed separately on the page.
 Several options may be listed for an item with instruction to please circles those conducted.
 A notes column will appear on the right side of the page clearly distinct from the checklist.
 A diagram of all phases will appear on the lists with the particular phase emphasized.
 A final “go” or “no-go” item to check may appear on some lists.
 Date of list creation and/or revision will appear on all lists.
 The initial lists in this paper will be considered prototypes as they have not yet been validated
with simulated or actual frontline users.
 Lists will be updated to suit current user needs.
Literature Review: Systems Design
Design Methodologies used to create the Checklists
The checklists will be populated by tasks taken from the Software Development Lifecycle (SDLC)
(Turban, 2009) and other methodologies including project management, Interaction Design and User-
Centered Design (UCD). The complete range of tasks will be divided into separate shorter lists based on
the phases of the SDLC. This is a logical division as the SDLC phases “consist of sequential processes by
which information systems are developed” (Turban, 2009). My focus in this paper is on the Systems
Investigation (and Feasibility Study), Analysis and Design phases and these are the phases I will describe in
detail and create Checklists for. I will also include a checklist of general System Requirements which is
relevant to the Design phase.
The SDLC Waterfall and Iterative Waterfall Models
A traditional approach to the SDLC, the waterfall model provides a helpful framework for
understanding strategic sequencing of activities. This sequencing structure is essential for complex

projects and enhances the chance of success on almost all projects. Feasibility of project success is
investigated upfront. The Feasibility Study asks questions such as: Is there passionate buy-in from top
level leaders who can allocate and protect investments of time and money? Does the project align with
organizational and IT strategy?
The “Iterative” Waterfall model differs from a traditional waterfall in that it allows for quicker
movement through the phases with the understanding that the project will cycle around through the
phases repeatedly. There is no expectation for getting things exactly right the first time around. Iterative
models “define an initial list of user requirements, builds a prototype system, and improve the system in
several iterations based on user feedback.” (Turban, 2009) This approach speeds up development and
allows users to clarify needs they may not have been able to articulate upfront. Many of the projects I’ve
led have used this model. This looser structure is more feasible when project scale is smaller and free or
low cost web-based tools are chosen.
Iterative Waterfall Model of Systems Development. This paper focus on the phases of feasibility, analysis
and design.
Achieving Discipline without Being Overly Prescriptive
Certain methods and activities of design are more useful than others for particular problems.
Many of the sources on design discuss problems with highly specific, restrictive, centrally imposed
methodology. Such top down prescriptive methods do not fit environments full of complexity, creativity,
unpredictability, expertise and individual temperaments and are usually rejected (Jefsioutine, Design
Frameworks, 2006). Dan Saffer argues that design is an applied art (Saffer, 2010) and other authors argue
that less prescriptive frameworks can create a principled context where key basic purposes are covered
using a range of tools and methods from various disciplines (Jefsioutine, Design Frameworks, 2006).

Organization of the Design Checklists
The rest of this paper contains systems design tasks descriptions and related information
correlated with the tasks contained in the actual checklists. The actual checklists follow the descriptive
sections. Description and checklist task numbering is correlated for referencing purposes. Tasks are
divided and named using phases from the SDLC. Page breaks occur between phases.

Systems Investigation and Feasibility Analysis
1. Initial Framing of the Problem: The initial stage in the SDLC is system investigation. Time is well
invested here in a preliminary “framing” of the system problem. Information systems are
embedded within a larger context of organizational context, procedures, training, etc. Zooming
out and establishing a broader holistic view of the existing system can help clarify what the actual
problem is that needs to be solved. Further framing in greater detail will occur again in the
Analysis phase.
2. Project Priority: The project should be prioritized according to its alignment with mission goals,
the scope of its potential impact and the stakeholders it will benefit.
3. Feasibility Study: The feasibility study is the primary task of the Systems Investigation phase. It
answers the “go-no go decision” e.g. “Should we move forward with this project?” In order to be
feasible and worthy of investment a new or modified system project must meet feasibility criteria
in the following four areas (Turban, 2009):
a. Technical Feasibility: The system must work with available hardware, software and
devices used currently and projected to be in place in the future per the IT strategy of the
organization (see Organizational Feasibility below).
b. Economic Feasibility: The system must meet time and cost constraints. What is the
budget for technology, training or related sources that could be utilized? Can existing IT
resources be used?
c. Organizational Feasibility: “A design strategy that doesn’t work for the overall organization’s
strategy is like a bad organ transplant: the host body will reject it (Saffer, 2010).” System must
align with organizational and IT strategy and meet legal and other constraints (Turban,
2009). Documentation including websites, business plans, constitutions, mission
statements, IT strategy plans etc. can be reviewed and should be confirmed by discussion
with appropriate personnel.
d. Behavioral Feasibility: System must engage stakeholders and users who generally fear
change and commonly sabotage or avoid new systems.
i. Did the Initial Concept Come from users? (Oregon State Computer Engineering
Dept., 2000)
ii. Staff Chart: Gaining access to a staff (organizational) chart is a good way to start
examining behavioral issues in this phase and in system analysis and design
phases. It can provide a framework for organizing information and identifying
stakeholders. Go over the chart with someone to be sure it is up to date and you

incorporate informal dynamics.
iii. Top Leadership Support and Modeling: Scott Anthony, David Duncan and Pontus
M.A. Siren state in their Harvard Business Review article on innovation that
“Every project… should have a senior executive sponsor or champion who
believes in it deeply” (Scott Anthony, 2014). They call this a “shepherding
function” essential to preventing new projects from being smothered by the
demands of organizational life (Scott Anthony, 2014). Case studies have revealed
that to foster the success of information related processes “people must be
aligned with a vision and… the alignment must be from the top down… leaders
must… lead by example.” (Plessis, 2007)
iv. Dual value proposition for the user: User engagement with a system often
requires that the system “add value to staff’s everyday working environment….
Successful enterprises ensure this dual value proposition.” (Plessis, 2007)
v. Trust: Users must trust the system.
vi. Dedicated system staff member(s) with time to devote: Dedicated staff roles are
an information system success factor as they can help insure the process is
managed consistently in a structured way. (Plessis, 2007)
vii. Time and Training: How much time is available for each project phase including
ongoing maintenance? Who will perform tasks? Will users devote time to give
input, undergo training? Do formal training processes or personnel in charge of
training in administrative or technology areas exist in the organization and will
training be supported by top leadership?
viii. What can we learn from past successes or failures? What would it take to make
this wildly successful?

Systems Analysis 1 of 2
Deliverable: System Requirements Hypothesis
(This Hypothesis Combined with the Deliverable of Analysis 2 (Stakeholder Research Findings) leads
to the creation of the Design Brief with System Requirements which is the deliverable of the complete
Analysis phase (parts 1 and 2 combined). Analysis is actually one phase but I have broken it into two
phases for the sake of organization of tasks. They do not occur strictly sequentially and you move back
and forth between them.)
If the feasibility analysis has led to a “go” decision, the project can justifiably move into the analysis
phase. Generally, the closer the involvement of the designer(s) with users, the better the chances of
creating a system that will meet user needs. The systems analysis phase of the SDLC involves:
“examination of the business problem in more detail… processes involved [include]… gathering
information about the existing system…” and delivering a set of requirements for the new system.
(Turban, 2009)
1. Determining Your Design Approach: Design approach for a project should be consciously chosen
and the best approach varies by project. The best designers know when to use each approach so
their solutions are holistic. Three approaches are described by Dan Saffer in the book Interaction
Design (Saffer, 2010) including Rapid Expert, Activity-Centered and User-Centered Design. Rapid
Expert Design is how most design occurs given limited resources. If the designer has enough
experience and the problem is familiar enough or simple enough this may be the way to go and
involves less analysis work. Activity-Centered design focuses on tasks and does not zoom out to
consider the problem from a larger perspective. User-Centered Design involves users from the
onset with lots of interaction and co-creation. Users guide requirements and preferences while
the designer facilitates translation of user goals into the design of the system (Saffer, 2010).
Depending on where you decide to go on this continuum of approaches will inform your choice of
tasks from those described below.
2. Framing the Problem: Zooming out can help you clarify what the actual problem is that needs to
be solved. Systems are embedded within the context of processes, training and other realities.
Some problems have fuzzy borders, many stakeholders and no immediately clear solution.
3. Project Timeframe: Saffer recommends project planning by setting an end date. Then “work
backwards… blocking out time in chunks for the various portions of the design process. The plan
should be… posted somewhere where the team can see it… Key dates and deliverables should be

made known and agreed to by the team.” (Saffer, 2010). The plan will be revised but can help
clarify things.
4. Conducting Traditional Research and Analysis Independently: Research can be done
independently. Much of these research findings will need to be confirmed and brought to life by
discussion with stakeholders using the tasks in the Analysis 2 Phase.
a. Document Analysis: webpages, brochures, flyers, forms, procedures, manuals, receipts,
and other documents can be reviewed.
b. Analytics: Review system or site analytics on page visits, visitor flows, users, login
patterns, transaction statistics and receipts and other quantitative data.
c. Information requirements: Information requirements include information needed by
whom, when and in what format (Turban, 2009).
i. Information Inputs and Outputs Table (Input, Personnel, Process Activities,
Outputs) (upfront, recurring).
d. Workflow Analysis. Draw your own and have users draw them as well and improve on
yours. Discuss the drawings and consider emotional elements.
5. Understanding the Market (Subject Area): Trade journals and white papers as well as
investigation into other businesses in the same market can help a designer get their bearings
(Saffer, 2010). What are the best practices recommended? Is another organization has the
process just right and is willing to educate or partner.
6. Usability Evaluation of Current and Competitive Systems: An understanding of classic design
principles goes a long way toward spotting usability issues. The following are some key concepts
to factor as well as a description of heuristics based evaluation which alone can produce major
insights.
a. Heuristic based evaluation is used and is defined as the judging of user-interface
compliance with key usability principles which have been validated (Nielsen, 1995). The
ten principles include System Condition visibility, relationship between system state and
reality, error prevention, aesthetics and minimalist planning, user ability to recognize,
diagnose and correct for errors, authentication, help, swift processing for both
experienced and novice users, familiar language and consistent visual cues (Nicola
Convertini, 2004). The three day heuristic evaluation conducted in the article from the
University of Bari (Nicola Convertini, 2004) provides an example of rapid design which
delivered significant value. This has relevance given the frequent constraints placed on
the design process in areas including time, money, expertise and coordinated
organizational support for design work (Saffer, 2010).

b. Design Principles to Supplement the ten Heuristics:
i. Visibility, and Feedback.
ii. Minimal Learning Curve and Existing mental maps : Conceptual models of
products allow users to make ongoing decisions without reference to instructions
by transferring “old knowledge to the new object” (Norman, 2002)
iii. Affordance: Affordance is defined as “the perceived and actual properties for the
thing, primarily those … that determine just how the thing could possibly be
used… Affordance provides strong clues to the operations of things.”
iv. Constraints: Constraints occur when “the world restricts allowed behavior” and
thereby the amount of knowledge in the head required for a particular situation
is reduced (Norman, 2002).
v. Permissions: Users should not be able to delete or irrevocably damage a
document. User permissions should be granted, updated and revoked
accordingly.
vi. Minimal Written Instructions: The above design techniques can be applied to
create a user experience environment that requires minimal written instruction.
Most of the knowledge is “in the world”, creating an intuitive experience
(Norman, 2002).

Systems Analysis 2 of 2
Deliverable: Design Brief with System Requirements
(Requirements must include 1. Strengths and weaknesses of existing system, 2. Functions
new system must have to solve the business problem and 3. User information requirements. The
Design Brief is created using the deliverables of both Analysis sections.)
1. Stakeholder Research: Stakeholder Research Complements and Improves Independent Research
and Analysis. As described by Shelley Evenson of Carnegie Mellon School of Design “ethnographic
methods can account for the complexity of service elements that are onstage, backstage, visible
and invisible in the service experience” (Saffer, 2010).
a. Determine Important Stakeholders: Find organizational charts and create a list of key
stakeholders you can talk to and survey. Who are key clients, users, leaders, consultants,
passionate about this issue? Determine this so you can leverage the information.
b. Partner with another researcher: “A second pair of eyes, ears and hands is immensely
valuable for observing, listening and recording, and for discussing and analyzing the
research data afterwards.” (Saffer, 2010)
c. Test Plan: A Test Plan should be devised containing neutral questions as well as test
routes “through the product that test functionality” (Saffer, 2010). Cognitive and
behavioral measures like “time taken to complete a task or the number of errors or
deviations from a critical path” can be recorded (Jefsioutine, 2006).
d. Stakeholder Observation and Interviews: (Go to them, talk to them): Don’t make subjects
come to you. Get out of the comfort of your office and observe activities in the
environment they are typically performed in. Observational and interviewing strategies
include “Fly on the wall”, shadowing (with or without asking questions along the way)
directed storytelling, role play, interviewing those not involved in the process, asking for
a tour of a subjects desk/computer etc. Have subjects tell their own stories in their own
manner directly to designers so that nuances can be captured. Activities can also include
asking subjects to draw or collage the life cycle of their experience and explain their work
when done. Ask them to explain the drawing but don’t tell them yin advance that yuou
will do this asking.
i. See Appendix for Stakeholder Interview Questions
e. Write stuff down: Capture with a camera, pen and pad etc. either during research or
directly after.

f. A summative “Opportunity Report” should record observations and patterns showing
trouble spots and suggestions for improvement (Saffer, 2010) can be created to inform
the design going forward. Analyze user mental models of current task structure (Oregon
State Computer Engineering Dept., 2000).
2. Surveys. Surveys have been described in detail already in the section on evaluating outcomes.
Both of these surveys appear in the Appendix of this paper.
a. System Usability Scale (SUS)
b. Supplementary Questionnaire
c. Analyze survey results
3. Create a Design Brief Based on the Research Tasks Above: This document is an initial analysis of the
problem with initial suggestions for solutions, technical constraints, timetable, deliverables, goals,
major stakeholder contact info etc. It is produced from research, interviews, competitive analysis
and used iteratively for capture and communication of information and further questions. For
system design, this can be the vehicle for presenting the deliverable of the Systems Analysis
phase which are the System Requirements.
a. Does the Brief present information in easy to understand visual format?
b. The following two tasks come from the Checklist for User-Centered Design (Oregon State
Computer Engineering Dept., 2000):
i. Did real users review and evaluate the design brief/initial design?
ii. Did designers clarify user feedback to arrive at a significant majority opinion?
c. The System Requirements presented with the Design Brief: The requirements should
include the following three elements (Turban, 2009):
i. Strengths and weaknesses of existing system
ii. Functions new system must have to solve the business problem
iii. User information requirements

Systems Design
Deliverable: Technical Design Specification
Systems Design considers Logical Design and Physical Design. Logical design includes what the
system needs to do in abstract terms including “inputs, outputs, processing, controls, security and IS
roles”. Physical Design which includes how the system performs using specific hardware, software and
procedures. Also included are interface design and a blueprint of how everything works together
(Turban, 2009).
Logical Design
1. Consider all touch points: Consider all points at which users will interact with the system including the
actual product and all related services, advertisements, referrals, support experiences, follow up contact
points etc.
2. Employ Usability Heuristics and Design Principles: “There is nothing more annoying than talking to
someone who doesn’t respond” (Saffer 131). See Analysis section for details.
3. Personnel Roles: Information Systems and other personnel involved in the system operation have
been defined.
4. Procedures and controls
a. Permissions and security
b. Collaborative Process best practices: Precise role definition: According to Lynda Gratton’s
article in the Harvard Business Review, “Cooperation increases when the roles of
individual team members are sharply defined yet the team is given latitude on how to
achieve the task”, Face time: The fostering of informal relationships and accountability
(Erickson, 2007).
Physical Design
5. Tool selection and acquisition based on system requirements: Internet based software is more and
more common and is sometimes called “cloud computing” or Software-as-a-Service (SaaS).
Vendors host applications and customers have no control over the software and access it over a
network, usually the internet. (Turban, 2009) Most system administrators feel that increasing
reliance on cloud computing is inevitable and the majority of businesses are adopting cloud
computing. (Axelrod, 2013) “There’s no need to huddle around the same computer of send files
back and forth over email if you want to collaborate with other people” stated the website
howtogeek.com. (Hoffman, 2014)
a. Typical System Requirements Considerations Checklist. This is one of the checklists at the
end of this paper.

b. Add Specific System Requirements (deliveable of Analysis Phase) to General considerations
c. Vendor Rating Tool worksheet: Not part of this paper.
d. Market Research: The identification of software alternatives often involves research using
trade journals, consultants experienced in the area, surveying peers in similar companies
and web research. (Turban, 2009) . Industry Analysis conducted by companies like
Gartner, the world’s leading IT research and advisory company present information on
factors such as market share, fit for various environments, sector adoption rates,
functionality, infrastructure and others.
6. Create the Technical Design Specification Deliverable: From the logical and physical designs and
the steps below are completed (Turban, 2009).
a. Both logical and physical design are reviewed with all appropriate stakeholders.
b. Did users specifically notice that their suggestions had been incorporated? (Oregon State
Computer Engineering Dept., 2000)
c. Did users specifically say the new design was better? (Oregon State Computer
Engineering Dept., 2000)
d. Revise the Specification as needed.
e. All appropriate stakeholders approve the specifications and formally acknowledge that
they are being now “frozen” to prevent scope creep leading to runaway projects over
budget and/or deadline or abandoned entirely.
7. The Frozen Specifications are the Deliverable for the build/development phase.

Development, Testing, Implementation, Operation and
Maintenance Phases Will Not Be Explored in Depth
These subsequent crucial phases are not within the focus of this paper but I will summarize them
briefly to place the other phases in context.
Development
In the development phase, depending on tool selection choices there will be varied degrees of
coding and customization required. Even Software-as-a-Service (SaaS) and Off-the-Shelf applications
require customization. I have personally used many SaaS applications including Google Apps, Wufoo
forms, content management systems and email marketing software. Many of these applications are very
flexible, providing you the tools to create applications according to your specifications without much if
any actual programming.
Testing
Testing includes both technical and functional testing by developers and user testing. I will not
review this topic again as it has been elaborated already in the previous “Criteria and Methods of
Checklist Evaluation” section of this paper.
Implementation
Implementation can be conducted via direct conversion (all at once) to the new system, pilot
conversion or phased conversion (to allow for evaluation before full direct conversion). Training is very
important to the success of the system and can occur through various means.
Operation and Maintenance
These phases are essential and include technical fixes, updates and the addition of new
functionality. They often use the most resources of all in the lifecycle of the system.
The Design Checklists
I’ve created five design checklists for this paper corresponding to the phases of Systems
Investigation/Feasibility Analysis, Systems Analysis (Part One), Systems Analysis (Part Two), Systems
Design and System Requirements. The Lists appear below. They are prototypes in need of use, testing
and validation.

SYSTEMS DESIGN CHECKLIST Revised 3/16/15 by Adam Wilson
> INITIATION > SYSTEMS INVESTIGATION/FEASIBILITY > SYSTEMS ANALYSIS
1 > ANALYSIS 2 > DESIGN > PROTOTYPE-TEST > REFINE > IMPLEMENT > OPERATE >
NOTES
 1. The system problem has been framed to clarify where the
root of the actual problem lies.
 2. This project has been ranked a top priority against others
based on mission goals, beneficiaries and scope of impact.
 3a. This project has been deemed technically feasible.
 3b. Resources are available which make this project
economically feasible.
 3c. This project aligns with organizational and IT strategy
based on documentation and confirmation with appropriate
personnel.
 3di. The initial concept or motivation for this project came
from users or users have voiced a willingness to make
changes to support this project.
 3dii. The organizational staff chart has been obtained and
reviewed to identify various project stakeholders.
 3diii. A top leader with adequate authority is passionate
about the project and has agreed formally to shepherd it
through all phases including research, design, build, test,
iterate, launch, training, operation. Who?
Who is passionate about this?
How to engage them?
 3div. Stakeholders have definitely been able to see potential
value to their own everyday work life from this project.
 3dv. Users trust of information systems and the leaders of
this project enough that they will trust the system with their
valuable information resources.
 3dvi. A dedicated, qualified staff member has been allocated
by top leadership to take ownership of this project.
 3dvii. Formal training processes and personnel exist and have
been authorized to insure proper training on the new system.
 3dviii. Past failures and successes have been shared by users
and stakeholders and this project involves factors that
predict it will succeed and can mitigate risks.
 DELIVERABLE: GO OR NO-GO DECISION (please circle)
1. Will not proceed based on these criteria (list):
2. Looks promising, will be deferred for Review on:
3. May proceed, given the following additional criteria are met:
4. Will proceed beginning:

> INITIATION > INV./FEASIBILITY > SYSTEMS ANALYSIS 1 of 2 > ANALYSIS 2 > DESIGN >
PROTOTYPE-TEST > REFINE > IMPLEMENT > OPERATE >
NOTES
TRADITIONAL RESEARCH
 1. Have a design approach been consciously chosen?
 2. The system problem has been framed to clarify where the
root of the actual problem lies.
 3. Tentative Key dates and deliverables have been agreed to
by the team and the tentative timeline has been posted.
 4a. Key documents have been identified, collected and
analyzed.
 4b. Have you reviewed relevant quantitative information
such as analytics, user logins or transaction statistics?
 4c-d. A hypothetical workflow analysis chart and/or table of
information inputs, people, process, and outputs has been
created to go over with stakeholders.
 5. Have you reviewed trade journals, industry analysis and
competing businesses and identified best practices?
USABILITY and DESIGN
 6a. Have designers conducted a personal heuristic evaluation
of the existing system?
 6b. Designers have evaluated the existing system using
design criteria, identified issues and recorded ideas for their
correction.
 DELIVERABLE: System Requirements Hypothesis

> INITIATION > INV./FEASIBILITY > SYSTEMS ANALYSIS 1 > SYSTEMS ANALYSIS 2 of 2>
DESIGN > PROTOTYPE-TEST > REFINE > IMPLEMENT > OPERATE >
NOTES
STAKEHOLDER RESEARCH
 1a. Have you discussed the organizational chart or similar
document to identify key stakeholders?
 1b. A second researcher has been recruited for a second
pair of eyes and discussion of findings.
 1c. A Test Plan has been devised containing neutral
questions for stakeholders and planned research
activities.
See stakeholder interview questionnaire.
 1c. A test route through the product has been designed
“through the product in order to test functionality.
 1d. Users have been observed in their own environments
with neutral questioning and recording with text, photos,
etc. The following activities have been conducted (please
conduct at least 2 and circle them at right).
Fly on the wall, shadowing, directed
storytelling, role play, interviews, tour of
computer and workflow etc., user workflow
drawings explained afterward.
 1f. A summative “Opportunity Report” has been created
from observations and identified patterns.
 2a. The System Usability Scale (SUS) or alternate usability
questionnaire has been administered and scored.
SUS Supplementary Questionnaire available.
 2c. Questionnaire results have been analyzed.
3. DELIVERABLE (OF ANALYSIS 1 & 2): Design Brief with System Requirements
 Stakeholder Research analysis has informed the Design Brief.
 Traditional Research hypothesis have informed the Design Brief.
 Information has been structured in a visual and easy to understand way in the Brief.
 Strengths and weakness findings about the existing system are contained in the brief.
 Functions the new system must have are included.
 User information requirements are included.
 Has a meeting been scheduled with appropriate stakeholders including real users to go over the Brief
 Did designers clarify user and stakeholder feedback about the Brief to arrive at a significant majority opinion,
revise and agree on the system requirements and other elements of the Brief.

INV./FEASIBILITY > ANALYSIS 1 > ANALYSIS 2 > SYSTEMS DESIGN > PROTOTYPE-TEST > REFINE >
IMPLEMENT > OPERATE >
NOTES
LOGICAL DESIGN
1. All touch points at which users will interact with the system
including related services have been mapped holistically.
 2. Usability Heuristics and Design Principles have been
applied to each interface and aspect of the system.
 Information Systems and other personnel involved in the
system operation have been defined.
 Appropriate user permission profiles have been created.
 Security measures both behavioral and technical have been
designed for information backup and access.
 Collaborative issues such as precise role definitions, face
time and accountability have been designed.
PHYSICAL DESIGN
 System Requirements from the Analysis phase have been
added to the general system requirements table.
See table in Appendix.
 Market research has identified software alternatives.
 Potential vendors have been rated using Requirements table Consider using the Vendor Rating Tool
worksheet.
DELIVERABLE: FROZEN TECHNICAL DESIGN SPECIFICATIONS
 Both logical and physical design have been reviewed with all appropriate stakeholders including users.
 Did users specifically notice that their suggestions had been incorporated?
 Did users specifically say the new design was better?
 Specification have been revised as needed.
 All appropriate stakeholders have approved the final specifications and formally acknowledged they are being
“frozen” to prevent scope creep.

Generic System Requirements
Checklist
Taken from Introduction to Information Systems:
Supporting and Transforming Business (Turban, 2009)
Software Under Consideration:
_____________________
 Project specific requirements have been added to the generic requirements below.
 1. Functionality
 2. IT Strategy Alignment
 Usability
 Availability of Quality Documentation and
Training Resources
 Vendor Reputation, Industry Analysis
 What do competitors use?
 Cost. Free Trial?
 Integration with existing applications
 Necessary Hardware, Software and Networking
Resources and compatibility:
 Implementation, Updates
 Security
 usable for other processes
 Long term product availability
Revised 3/16/15 by Adam Wilson

Reflection
Surprising Finding
The single most surprising thing I begin to see in the 12 sources and other research on this
project is the power of simple, precise interventions to dramatically impact extremely complex situations.
I was pleasantly surprised About the Checklists I’ve been using for Administrative Work
My prior work in administration has been highly checklist driven. Although I had not yet studied
checklist best practices the lists I created for various aspects of my job follow some of them. They include
easy to miss and critical items, fit on one page, are quickly reviewable and ordered sequentially and/or
logically, were used collaboratively by administrative staff to track and communicate among team
members and were online to facilitate sharing and frequent updates (Katherine Radeka, 2015).
Comprehensive instructions were hyperlinked to items to facilitate access without overloading the
checklist itself. The lists were built in Google Sheets and the Design Checklists at the end of this paper
may eventually be migrated there as well. My office management checklists helped make me a very
reliable and consistent administrator and I would like to expand that kind of competency into the slightly
less routinized and more creative space of design.
The Checklists in the Paper are Prototypes
These checklists have not been trialed and validated with front line users either in a simulation or
in a real situation and revised as needed. They are merely prototypes.
Submitting the Checklists to Design Scrutiny
While the design checklist should be subjected to the best practice standards of creating checklist
they should then again be subjected to the best practice standards of design. Under such analysis I
observe that I prefer online checklists which can be shared and accessed remotely. I suspect the next
iteration of them will involve migrating them to Google Sheets.

References
Aline Chevalier, M. Y. (2002). We site designs: Influences of designer's expertise and design constraints.
International Journal of Human-Computer Studies, 57-87.
Allen, D. (2002). Getting Things Done: The Art of Stress-Free Productivity. Penguin Books.
Asaf Degani, E. L. (1991). Human factors of flight-deck checklists: The normal checklist. Retrieved from
NASA Technical Reports Server (NTRS): http://ntrs.nasa.gov/search.jsp?R=19910017830
Axelrod, G. (2013, October 23). 47 Stats You Need to Know About the Google Apps Ecosystem. Retrieved
from bettercloud.com: http://blog.bettercloud.com/google-apps-stats/
Brooke, J. (1986). SUS - A quick and dirty usability scale. Farley, UK: Redhatch Consulting, Ltd.
Coleman, M. (2015). Process. Retrieved from megancoleman.com:
http://www.megancoleman.com/process/
davidallengtd.com. (2014). Getting Things Done Daily Planner. ataglance.
Erickson, L. G. (2007). Eight Ways to Build Collaborative Teams. Retrieved from Harvard Business Review:
http://www.harvard-
samsung.net/hbspCourse/hmm11/team_leadership/base/resources/EightWays_BuildCollaborativ
eTeams.pdf
Gawande, A. (2009). The Checklist Manifesto. New York: Metropolitan Books.
Geracie, G. (2010). Take Charge Product Management. Actuation Press.
Hoffman, C. (2014, February 23). How to Collaborate on Documents Over the Internet. Retrieved from
howtogeek.com: http://www.howtogeek.com/183176/how-to-collaborate-on-documents-over-
the-internet/
Jefsioutine, J. K. (2004). Designing in the E-Society: The Experience Design Framework. Proceedings of the
IADIS International Conference (pp. 1103-1107). Avila, Spain: e-Society. Retrieved February 1,
2015, from
http://www.mgt.ncu.edu.tw/~ckfarn/doc/conference/IADIS%20ES2004_Vol2.pdf#page=415

Jefsioutine, J. K. (2006). Design Frameworks. Encyclopedia of Human Computer Interaction. (C. G. (ed),
Ed.) University of Central England, UK: IGI Global. Retrieved February 1, 2015, from
http://common.books24x7.com.ezproxy.depaul.edu/toc.aspx?bookid=14703
Katherine Radeka. (2015). A Checklist for Designing a Checklist. Retrieved from
leantechnologydevelopment.com:
http://leantechnologydevelopment.com/system/files/Checklists-Letter.pdf
Katherine Radeka, Whittier Consulting Group, Inc. (2015). A Checklist for Designing a Checklist. Retrieved
from leantechnologydevelopment.com:
http://leantechnologydevelopment.com/system/files/Checklists-Letter.pdf
Lewis, J. R. (1995). IBM Computer Usability Satisfaction Questionnaires. International Journal of Human
Computer Interactions, 57-78.
McKeown, B. J. (2013). Assessing and manageing risk with people with physical disabilities: the
development of a safety checklist. Health, Risk and Society, 162-175.
Nicola Convertini, A. M. (2004). Usability Models on Institutional Portals: A Case Study at The University of
Bari., (pp. 1099-1102).
Nielsen, J. (1995, Janurary 1). 10 Usability Heuristics for User Interface Design. Retrieved from
nngroup.com: http://www.nngroup.com/articles/ten-usability-heuristics/
Norman, D. A. (2002). The Design of Everyday Things. New York: Basic Books.
Oregon State Computer Engineering Dept. (2000). Checklist for User-Centered Design. Retrieved from
web.engr.oregonstate.edu: https://web.engr.oregonstate.edu/~pancake/cs252/checklist.html
Palmius, J. (2007). Criteria for measuring and comparing information systems. Proceedings of the 30th
Information Systems Research Seminar in Scandinavia.
Plessis, M. d. (2007). Knowledge Management: what makes complex implementations successful? Journal
of Knowledge Management, 91-101.
Saffer, D. (2010). Designing for Interaction: Creating Innovative Applications and Devices. Berkeley: New
Riders.
Scott Anthony, D. D. (2014, December). Build an Innovation Engine in 90 Days. Harvard Business Review.

Turban, R. K. (2009). Introduction to Information Systems. Hoboken: Wiley and Sons.
usability.gov. (2015). System Usability Scale (SUS). Retrieved from usability.gov:
http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
Warning, P. (2009). Information Seeking and Stoppping Among Undergraduate Interns. Proceedings of the
2009 INternational Conference on Knowledge Management. Hong Kong: S.K.W.
World Health Organization. (2009). Surgical Safety Checklist. Retrieved from who.int:
http://www.who.int/patientsafety/safesurgery/checklist/en/

Appendix A
System Usability Scale
Instructions: Use after the respondent has had an opportunity to use the system being evaluated, but
before any debriefing or discussion takes place. Respondents should be asked to record their immediate
response to each item, rather than thinking about items for a long time. All items should be checked. If a
respondent feels that they cannot respond to a particular item, they should mark the center point of the
scale.
Strongly Strongly
Disagree agree
1. I think that I would like to use this system
frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a
technical person to be able to use this system
5. I found the various functions in this system
were well integrated
6. I thought there was too much inconsistency in
this system
7. I would imagine that most people would learn
to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could
get going with this system
Scoring the SUS
SUS yields a single number representing a composite measure of the overall usability of the system being
studied. Note that scores for individual items are not meaningful on their own. To calculate the SUS score,
first sum the score contributions from each item. Each item's score contribution will range from 0 to 4.
For items 1, 3, 5, 7, and 9 the score contribution is the scale position minus 1. For items 2,4,6,8 and 10,
the contribution is 5 minus the scale position. Multiply the sum of the scores by 2.5 to obtain the overall
value of SU. SUS scores have a range of 0 to 100.
© Digital Equipment Corporation, 1986.

Appendix B
Stakeholder Interview Sample Questions
Taken from Interaction Design (Saffer, 2010)
1. Who are you and what is your role in the org
2. Why is this project important to you? To the org? (internal context)(look for unstated
goals/agendas)
3. What would make a successful project? (metrics)
4. Has anyone ever tried to address this problem before? (Why?)
5. What doesn’t this project cover? (Why?)(scope)(What are constraints, boundaries can’t cross?)
6. If we could only do only one thing with this project, what would that be? (Why?)
7. How could this project affect your day-to-day life? (Organizational expectations, part of life at…)
8. Are there any issues about this project I should be aware of?
9. What are the risks in doing this project? What could make it fail?
10. What are your competitors doing in this space?
11. “What would customers do (instead) if all the traditional competitors went away?
12. Who else should I talk to about this project?
I’ve added these questions taken from a Checklist for User-Centered Design (Oregon State Computer
Engineering Dept., 2000)
1. What are your work goals related to the project?
2. How do you currently structure those goals into tasks?
3. Why do you do it that way?
4. How do you wish you could do it?

Appendix C
Stakeholder Questionnaire Items Supplemental to the SUS
The following questions are not directly addressed by the SUS. They could be part of a separate
evaluation questionnaire with follow-up interactions. I have limited them to 25 per the recommendations
by SUS article author John Brooke (Brooke, 1986). They are designed to be answered using a Likert scale
from 1-5 strongly disagree to strongly agree.
Regarding expanding software choices for users and use on multiple devices.
1. The system is my preferred way to perform the tasks it is designed for.
2. The system has worked well on all my devices.
From A 1995 Computer System Usability Questionnaire developed by IBM (Lewis, 1995).
1. I can effectively (and efficiently) complete my work
2. Whenever I make a mistake I recover easily and quickly.
3. The system was intuitive or Instructions were visible or easily retrievable
4. List the most negative aspects: (3 open ended fields to complete)
5. List the most positive aspects: (3 open ended fields to complete)
The Experience Design Framework “includes attitude measurements and emotional response” as well as
social and pleasurable elements at the top of a “hierarchy of user needs” (Jefsioutine, 2006).
1. The system can help divide workload to the right people
2. I would definitely recommend this system to others
3. Working with this system would support positive relationships with others.
4. This system could help users work together well as a team.
5. This system could support improved relationships between different departments.
The article Criteria for Measuring and Comparing Information Systems (Palmius, 2007) contributes these
questions related to organizational and knowledge management impacts of the system.
1. The system will help the organization capture significant knowledge embedded in individuals.
2. The system makes this knowledge easy to access and use by the appropriate people.
3. Processes are in place for leveraging this knowledge to create value.
4. People are regularly using the processes provided by the system for knowledge capture.
5. The system provides strong ROI.
6. The system significantly enhances the competitive advantage of the organization.
7. The system significantly enhances the satisfaction of external customers.
Additional Questions regarding trust and sustainable operation and support.
1. I am very comfortable that critical information in the system is safe.
2. Updating the system with information to keep it very current will be quite feasible.
3. We have the right people to maintain the system in good working order long term.
4. Problems with the system are addressed rapidly to my satisfaction.
5. My satisfaction and/or effectiveness with the system would be very much enhanced by these
changes or enhancements (open ended question):

Adam Wilson AP Final Project REVISED FINAL DRAFT

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Viewers also liked

Viewers also liked (13)

Similar to Adam Wilson AP Final Project REVISED FINAL DRAFT

Similar to Adam Wilson AP Final Project REVISED FINAL DRAFT (20)

Adam Wilson AP Final Project REVISED FINAL DRAFT