Brian explains and presents case study material on how to measure intranet performance by observing people trying to carry out Top Tasks. - Usability practices involved - Clear reporting using simple measures - Continuous improvement approach

1. Copyright © Customer Carewords Ltd.
Brian Lamb
@blamb
Measuring Intranet
Performance - Task based
approach

2. Current
website
Customers
Stakeholders
Competitors,
strategies and
other documents
Long list Short list Survey to
uncover the
customers’ top
tasks
New website

3. Please look at the following list and choose ONLY the Top
FIVE Topics that are MOST IMPORTANT to you. Top 20
Tasks Total
% of
3795
Cumul
Vote
Cumul
Words
Grants / equity funding from Enterprise
Ireland (eligibility, description) 332 9% 9% 1%
Research and development funding 210 6% 14% 2%
Grants / equity funding online applications 205 5% 20% 4%
Market / competitor research 144 4% 23% 5%
Commercial funding sources (VC, seed,
business angels) 128 3% 27% 6%
Market entry strategy advice 120 3% 30% 7%
Introductions to key industry decision
makers / advisors 100 3% 33% 8%
EU funding for companies 99 3% 35% 10%
Find Enterprise Ireland people
(development, market, technical, legal) 99 3% 38% 11%
Tax relief (BES, seed capital) 91 2% 40% 12%
How-to guides 83 2% 42% 13%
Introductions to buyers (buyer feedback,
follow-up) 81 2% 45% 14%
Find an Enterprise Ireland office (HQ,
regional, international) 79 2% 47% 16%
Target customer / distribution channel
identification 74 2% 49% 17%
Trade fairs 73 2% 51% 18%
Mentoring 72 2% 52% 19%
International marketing and sales training 71 2% 54% 20%
Claim grant payments online 68 2% 56% 22%
Intellectual property (patent, trademarking,
copyright) 68 2% 58% 23%
Help setting up overseas offices 66 2% 60% 24%
Example data from enterprise-ireland.com with permission

4. Long Neck
0
20
40
60
80
100
120
140
160
180
200
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141 146 151
Carewords
Vote

5. The Kheel Center for Labor-Management Documentation and
Archives
https://creativecommons.org/licenses/by/2.0/
Observe users
trying to
complete
#TopTasks
Note
Success/Failure
and time taken
Use the Master Apprentice model
where users are the masters
and the observers are the apprentices

6. 45%
37%
45%
27%
39%
75%
75%
75%
75%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Task: find a person
News
Benefits
Leave
Overall Performance
Performance level
Metric
Top Tasks– Performance Dashboard

7. Management Model
Identify
top tasks
Measure
top tasks
Improve
Top
Tasks

8. Example Performance Dashboard
Metric Current Task
Performance
Target
Performance
Current improvement
actions
Task: find a
person
45% 75% Implementation of new
SharePoint solution
News 37% 75% Improving archive function
Improving links guidelines
Benefits 45% 75%
Leave 27% 75%
Search Not yet
measured
Progress against plan
Common
architecture

10. Step 2: Recruit Participants
Recruit 15-18
representative
people

11. Remote Measurement Sessions
1. Record session
2. “We are not measuring you”
3. Measure time for each task.
4. Observe problems and successes.
5. No “think aloud”, no discussions

12. Test representative tasks
Tasks Total Vote
% of Total
Vote 8730
Cumulative
Vote
Cumulative
Carewords
1
Find people (phone directory/book,
expert, employee photos) 915 10% 10% 1%
2
Pay / salary details, view your
paycheck, tax witholdings 818 9% 20% 3%
3 Board news (internal for employees) 815 9% 29% 4%
4
Benefits (enrollments, insurance
coverage, workers compensation) 453 5% 34% 5%
5
Leave and attendance (holidays,
maternity, sick, annual, overtime) 395 5% 39% 7%
6
Online applications (Oracle,
PeopleSoft, Discoverer, JP Morgan
Chase, PaymentNet, Travel Manager) 372 4% 43% 8%
7 NewsClips (from newspapers) 339 4% 47% 9%
8 Credit Union 282 3% 50% 11%
9 Shuttle schedule 282 3% 54% 12%
10 Policies and procedures 240 3% 56% 14%
11
Parking, carpools/vanpools, lottery,
passes 231 3% 59% 15%
12 Cafeteria menus 226 3% 61% 16%
13 Division news 221 3% 64% 18%
14
Calendars (events, meetings,
holidays, pay, training) 203 2% 66% 19%
15 Retirement (Thrift Plan, gifts, parties) 201 2% 69% 20%
16 Organization charts 183 2% 71% 22%
17 Forms, memo templates 154 2% 73% 23%
18 Visitors (tours, registration) 153 2% 74% 24%
Task ItB Find People poll result*
Find out which immediate section/unit XX works
in
1 – Employee’s immediate
unit/section
Find out which immediate section/unit XX works
in
1 – Employee’s immediate
unit/section
Find out who the Chief Financial Officer reports
to
3 – Place in organization chart
Find out who the Director of the Corporate
Affaires is
3 – Place in organization chart
XXis out of the office but you need to speak to
someone else on his team. Find another person
who works in the same immediate section/unit
4 – Who to contact if this person is
not around
Find the mobile phone number for XX 5 – Employee’s mobile phone
number
Find an expert in Web Usability 6 – Expertise
*Note: Poll result #2 was photo/picture

13. Find out which immediate section/unit
XX works in (top task #1)
 2010
 Sharepoint 6/2013
 Sharepoint 9/2013
(after changes)
80%
20%
Succes
s
Failure
Median time on
task = 44
seconds
63%12%
25%
Succes
s
Failure
Median time on
task = 57
seconds
100
%
Succes
s
Failure Median time on
task = 32
seconds

14. Find out which immediate section/unit
XX works in (top task #1)
 2010
 Sharepoint 6/2013
 Sharepoint 9/2013
(after changes)
Median time on
task = 35
seconds
100
%
Succes
s
Failure
Median time on
task = 14
seconds
100
%
Succes
s
Failure
68%6%
25%
Succes
s
Failure
Median time on
task = 38
seconds

15. Find out who the Chief Financial Officer
reports to (top task #3)
 Old ItB (2010)*
 Sharepoint 6/2013
 Sharepoint 9/2013
(after changes)
40%27%
Success
Failure
Disaster33%
Median time on
task = 73
seconds
38%
50%
12%
Success
Failure
Disaster
Median time on
task = 135
seconds
100%
Success
Failure
Disaster
Median time on
task = 120
seconds**
(45 seconds)
**”All sites” link causes
problems

16. XX is out of the office but you need to speak to someone else on
his team. Find another person who works in the same immediate
section/unit (top task #4)
 Old ItB (2010)*
 Sharepoint 6/2013
60%
7%
Success
Failure
Disaster
88%
12%
Success
Failure
Disaster
33
%
Median time on task =
45 seconds
Median time on task =
57 seconds

17. Find out who the Director of the Office of
Inspector General is (top task #3)
 Sharepoint 6/2013
Median time on
task = 122
seconds
38%
62%
Success
Failure
Disaster
Problems:
• Searching by division acronym “OIG” doesn’t help
• Mr. XXs title “Inspector General” led to confusion
• Section XX org chart in Sharepoint is useless

18. Find the mobile phone number
for XX (top task #5)
 Sharepoint 6/2013
100
%
Succes
s
Failure
Median time on task = 14
seconds

19. Find an expert in web usability (top task
#6)
 Sharepoint 6/2013 63%
12%
Success
Failure
Disaster
25%
Median time on task = 52 seconds
Problems:
• What does “Ask Me About” really mean?
Expertise?
• “All sites” tab causing problems
• Just not intuitive you can do this

20. Aggregate results
 Old ItB (2010)
 Sharepoint 6/2013
 Sharepoint 9/2013
(after changes)
63%
24%
Succes
s
Failure13%
67%
12%
Succes
s
Failure
21%
Median time on task = 52
seconds
Median time on task = 76
seconds
100
%
Succes
s
Failure
Median time on task = 55
seconds**
(25
seconds)
**”All sites” link causes
problems

21. What does this mean?
 What does this mean?
– People get slightly more accurate answers with Sharepoint,
but it takes them longer to find those answers. To vastly
improve the usefulness of Sharepoint and to make it easier
to Find People, we need to improve Sharepoint.
 So, how do we improve?

22. How to improve 1 – the easy stuff
 Simple, high impact improvements (what
worked in the September retests?)
– Explicitly labeling fields (Division, Section, Job Title,
etc)
 Mysite pages
 Search results pages
– Making it easy for people to search by both acronym
and full name
 Divisions
 Job titles
– Allow search by Division (full name and acronym) from
the main search box

23. How to improve 2
 Other suggestions
– Get rid of “All sites” button
– Improve trust in the org chart viewer
 Accuracy
 Design (who reports to whom?)
– Taxonomy must work well or people will
quickly lose trust in it
– Improve uniformity (job titles, dealing with
misspellings, section names, division names,
etc)

24. MEASURE
• Find out who the Chief Financial Officer
reports to (top task #3)
brian@customercareowrds.com
Image from: Ari Bakker http://www.sharepointconfig.com/2011/05/how-to-create-a-simple-
sharepoint-2010-people-directory/

25. How to improve 1 – the easy stuff
 Simple, high impact improvements (what
worked in the September retests?)
– Explicitly labeling fields (Division, Section, Job Title,
etc)
 Mysite pages
 Search results pages
– Making it easy for people to search by both acronym
and full name
 Divisions
 Job titles
– Allow search by Division (full name and acronym) from
the main search box