The document summarizes a 2007 study conducted by MITRE to investigate how MITRE employees use the company intranet and other IT tools. The study involved interviews with 36 employees, observations of 6 employees, and a survey completed by 537 employees. Key findings were analyzed and presented back to IT stakeholders and management. Lessons learned included streamlining the affinity diagramming process used to analyze qualitative interview data and using shorter observation sessions in the future. The study provided valuable insights into how tools were used and suggestions for improvements.

1. MITRE Study 2007
A Multi-Method Study at MITRE
UPA Boston Mini-Conference
28 May 2008
Bryn Dews
Marsha Nunes
© 2001 The MITRE Corporation. All rights reserved.
2
MITRE Background
Three Federally Funded R&D Centers (FFRDCs)

– Air Force (Defense and Intelligence)
– FAA (Aviation System Development)
– IRS (Enterprise Modernization)
Engineering environment

Lots of disciplines/specialties on sponsor projects

Heavy focus on System Engineering Lifecycle

model(s)
© 2003 The MITRE Corporation. All rights reserved.
My Background: BA  HCI
3
Hired as Business Analyst (Compensation &

Benefits) after 9+ years as BA/Manager for HR in
Banking
Team Lead on PeopleSoft Implementation

HRIT Manager

Team Lead for PeopleSoft in IT

Web Applications Project/Team Lead

Re-org opportunity to raise awareness for HCI

2003 – Team official – UCD Team Lead

– 2004 – add to staff
– 2005 – another add to staff
© 2003 The MITRE Corporation. All rights reserved.
1

2. 4
Purpose of 2007 MITRE Study
Investigate and observe how MITRE employees

use Intranet and other Corporate IT tools and
software
Focus Areas:

– Workspace and desktop setup
– Finding information
– Corporate Portal
– Phonebook
– Microsoft Outlook (email management)
– Awareness
Follow-on to Summer 2006 field study

© 2003 The MITRE Corporation. All rights reserved.
5
Methodology
Observations
(n=6)
Interviews (n=36)
Survey (n = 537/1260)
© 2003 The MITRE Corporation. All rights reserved.
6
Our Process: Overview
Create Project Plan

Hire Intern

IT stakeholder interviews (to narrow study

focus)
Structure study parts

Collect data

Summarize data

Present data to stakeholders and larger IT

community
© 2003 The MITRE Corporation. All rights reserved.
2

3. 7
Stakeholder Interviews
Development organizations

Top-Down interviews/meetings

Straw-man study outline

– Prior year study as starting point
– New ideas from usability testing
© 2003 The MITRE Corporation. All rights reserved.
8
Structure Study Parts
Interviews first

– Environment questions/observations
– Structured tasks
– Features used
– Catalog of applications
Observations next

Survey third

– Short and sweet
– Closed questions (corresponding to interviews)
© 2003 The MITRE Corporation. All rights reserved.
9
Collect Data
Interviews – Whole UCD Team

– Cross section (geographic, job level, job type,
organization)
– Template for notes
– Common terminology
– One hour format… loosely followed
Observations – three staff

Survey – reviewed by whole team

– Sr. Mgt. added open response question
© 2003 The MITRE Corporation. All rights reserved.
3

4. 10
Analyze/Summarize Data
Intern dissected quantitative responses in

interviews; put in tables; graphed
Team highlighted qualitative points in

interview/observation transcripts, created sticky-
notes
Affinity diagramming of notes for qualitative data

Survey Pro used to analyze survey data

Manual analysis of survey open response data

Project and Team Leads reviewed/fine-tuned final

report
© 2003 The MITRE Corporation. All rights reserved.
11
Read Out Study Results
Stakeholder organizations

– Sr. Management early
– Some came asking for it!
– Development organizations
– Loop back to other organizations, Chief
Engineers
COI sharing

Mostly small groups -> very interactive

© 2003 The MITRE Corporation. All rights reserved.
12
Lessons Learned
4 hour observations are tough sell

– Try more, shorter sessions next year
Smaller window = less data

Easier to schedule

Less tedious/invasive

Multiple facilitators make it harder to pick up on

trends across all sessions.
– Try more frequent team updates
– Review session notes as they’re written
Analyzing qualitative data in crunch always daunting!

© 2003 The MITRE Corporation. All rights reserved.
4

5. 13
More Lessons Learned
Affinity Diagram

– Worked well for analyzing qualitative data
But…
– Multiple people reading notes, adding to diagram
led to duplication
Streamlined process by reading and highlighting same

printouts
– Consistent session number labels would make it
easier to backtrack
– Good handwriting helps 
Overall well worth effort in terms of data collected

and outreach to MITRE community!
© 2003 The MITRE Corporation. All rights reserved.
14
Backup Slides
© 2003 The MITRE Corporation. All rights reserved.
15
Excerpts of Interview Checklist
PHYSICAL WORKSPACE
 What MITRE handouts & Giveaways are in/on their
workspace (PECE Cube, mugs)?
 Approx. monitor size & # of monitors?
 Screen Resolution?
 Any Assistive Technologies?
 Is SecurId present?
 For Sites, any noticeable work environment
differences?
© 2003 The MITRE Corporation. All rights reserved.
5

6. 16
Excerpts of Interview Checklist
OUTLOOK
 Are they using Tasks/Contacts and/or Notes?
 How many emails are in their inbox?
 How many email folders do they have?
 Are they using Rules to manage their email?
 Are they using flags to manage their email?
© 2003 The MITRE Corporation. All rights reserved.
17
Excerpts of Interview Catalog
FOR EACH SHOW A SCREEN SHOT OF THE APPLICATION OR
COLLECTION AND ASK:
How often do you use it?

Why?

How you get to it?

How did you learn about it?

NOTE: applications/sites included were pilots,
new/recently rolled/updated or additional features
in heavily used applications
© 2003 The MITRE Corporation. All rights reserved.
18
Excerpt of Observation Checklist
Desktop Browser
Monitor Size? Windows maximized?
Screen Resolution? Open/Close new
browser windows?
Windows maximized? Reuse browser
window? # of windows?
Number of apps open? Use Tabs? # of tabs?
Number of desktop Bookmarks? #?
shortcuts? Organization?
© 2003 The MITRE Corporation. All rights reserved.
6

7. 19
Survey Design
20 Questions (no kidding!)

Frequency of use (scale)

Ease of use (scale)

Appeal (scale)

Ever use (yes/no)

Which do you use (checkboxes)

Write in three improvements (short

answer)
Any other suggestions (long response)

© 2003 The MITRE Corporation. All rights reserved.
7