Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright © Customer Carewords Ltd.
Brian Lamb
@blamb
Measuring Intranet
Performance - Task based
approach
Current
website
Customers
Stakeholders
Competitors,
strategies and
other documents
Long list Short list Survey to
uncover ...
Please look at the following list and choose ONLY the Top
FIVE Topics that are MOST IMPORTANT to you. Top 20
Tasks Total
%...
Long Neck
0
20
40
60
80
100
120
140
160
180
200
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 ...
The Kheel Center for Labor-Management Documentation and
Archives
https://creativecommons.org/licenses/by/2.0/
Observe user...
45%
37%
45%
27%
39%
75%
75%
75%
75%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Task: find a person
News
Benefits
Leave
Overall...
Management Model
Identify
top tasks
Measure
top tasks
Improve
Top
Tasks
Example Performance Dashboard
Metric Current Task
Performance
Target
Performance
Current improvement
actions
Task: find a
...
Step 2: Recruit Participants
Recruit 15-18
representative
people
Remote Measurement Sessions
1. Record session
2. “We are not measuring you”
3. Measure time for each task.
4. Observe prob...
Test representative tasks
Tasks Total Vote
% of Total
Vote 8730
Cumulative
Vote
Cumulative
Carewords
1
Find people (phone ...
Find out which immediate section/unit
XX works in (top task #1)
 2010
 Sharepoint 6/2013
 Sharepoint 9/2013
(after chan...
Find out which immediate section/unit
XX works in (top task #1)
 2010
 Sharepoint 6/2013
 Sharepoint 9/2013
(after chan...
Find out who the Chief Financial Officer
reports to (top task #3)
 Old ItB (2010)*
 Sharepoint 6/2013
 Sharepoint 9/201...
XX is out of the office but you need to speak to someone else on
his team. Find another person who works in the same immed...
Find out who the Director of the Office of
Inspector General is (top task #3)
 Sharepoint 6/2013
Median time on
task = 12...
Find the mobile phone number
for XX (top task #5)
 Sharepoint 6/2013
100
%
Succes
s
Failure
Median time on task = 14
seco...
Find an expert in web usability (top task
#6)
 Sharepoint 6/2013 63%
12%
Success
Failure
Disaster
25%
Median time on task...
Aggregate results
 Old ItB (2010)
 Sharepoint 6/2013
 Sharepoint 9/2013
(after changes)
63%
24%
Succes
s
Failure13%
67%...
What does this mean?
 What does this mean?
– People get slightly more accurate answers with Sharepoint,
but it takes them...
How to improve 1 – the easy stuff
 Simple, high impact improvements (what
worked in the September retests?)
– Explicitly ...
How to improve 2
 Other suggestions
– Get rid of “All sites” button
– Improve trust in the org chart viewer
 Accuracy
 ...
MEASURE
• Find out who the Chief Financial Officer
reports to (top task #3)
brian@customercareowrds.com
Image from: Ari Ba...
How to improve 1 – the easy stuff
 Simple, high impact improvements (what
worked in the September retests?)
– Explicitly ...
Measuring intranet performance  - task based approach
Upcoming SlideShare
Loading in …5
×

Measuring intranet performance - task based approach

697 views

Published on

Brian explains and presents case study material on how to measure intranet performance by observing people trying to carry out Top Tasks.

- Usability practices involved
- Clear reporting using simple measures
- Continuous improvement approach

Published in: Business
  • Be the first to comment

Measuring intranet performance - task based approach

  1. 1. Copyright © Customer Carewords Ltd. Brian Lamb @blamb Measuring Intranet Performance - Task based approach
  2. 2. Current website Customers Stakeholders Competitors, strategies and other documents Long list Short list Survey to uncover the customers’ top tasks New website
  3. 3. Please look at the following list and choose ONLY the Top FIVE Topics that are MOST IMPORTANT to you. Top 20 Tasks Total % of 3795 Cumul Vote Cumul Words Grants / equity funding from Enterprise Ireland (eligibility, description) 332 9% 9% 1% Research and development funding 210 6% 14% 2% Grants / equity funding online applications 205 5% 20% 4% Market / competitor research 144 4% 23% 5% Commercial funding sources (VC, seed, business angels) 128 3% 27% 6% Market entry strategy advice 120 3% 30% 7% Introductions to key industry decision makers / advisors 100 3% 33% 8% EU funding for companies 99 3% 35% 10% Find Enterprise Ireland people (development, market, technical, legal) 99 3% 38% 11% Tax relief (BES, seed capital) 91 2% 40% 12% How-to guides 83 2% 42% 13% Introductions to buyers (buyer feedback, follow-up) 81 2% 45% 14% Find an Enterprise Ireland office (HQ, regional, international) 79 2% 47% 16% Target customer / distribution channel identification 74 2% 49% 17% Trade fairs 73 2% 51% 18% Mentoring 72 2% 52% 19% International marketing and sales training 71 2% 54% 20% Claim grant payments online 68 2% 56% 22% Intellectual property (patent, trademarking, copyright) 68 2% 58% 23% Help setting up overseas offices 66 2% 60% 24% Example data from enterprise-ireland.com with permission
  4. 4. Long Neck 0 20 40 60 80 100 120 140 160 180 200 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141 146 151 Carewords Vote
  5. 5. The Kheel Center for Labor-Management Documentation and Archives https://creativecommons.org/licenses/by/2.0/ Observe users trying to complete #TopTasks Note Success/Failure and time taken Use the Master Apprentice model where users are the masters and the observers are the apprentices
  6. 6. 45% 37% 45% 27% 39% 75% 75% 75% 75% 75% 0% 10% 20% 30% 40% 50% 60% 70% 80% Task: find a person News Benefits Leave Overall Performance Performance level Metric Top Tasks– Performance Dashboard
  7. 7. Management Model Identify top tasks Measure top tasks Improve Top Tasks
  8. 8. Example Performance Dashboard Metric Current Task Performance Target Performance Current improvement actions Task: find a person 45% 75% Implementation of new SharePoint solution News 37% 75% Improving archive function Improving links guidelines Benefits 45% 75% Leave 27% 75% Search Not yet measured Progress against plan Common architecture
  9. 9. Step 2: Recruit Participants Recruit 15-18 representative people
  10. 10. Remote Measurement Sessions 1. Record session 2. “We are not measuring you” 3. Measure time for each task. 4. Observe problems and successes. 5. No “think aloud”, no discussions
  11. 11. Test representative tasks Tasks Total Vote % of Total Vote 8730 Cumulative Vote Cumulative Carewords 1 Find people (phone directory/book, expert, employee photos) 915 10% 10% 1% 2 Pay / salary details, view your paycheck, tax witholdings 818 9% 20% 3% 3 Board news (internal for employees) 815 9% 29% 4% 4 Benefits (enrollments, insurance coverage, workers compensation) 453 5% 34% 5% 5 Leave and attendance (holidays, maternity, sick, annual, overtime) 395 5% 39% 7% 6 Online applications (Oracle, PeopleSoft, Discoverer, JP Morgan Chase, PaymentNet, Travel Manager) 372 4% 43% 8% 7 NewsClips (from newspapers) 339 4% 47% 9% 8 Credit Union 282 3% 50% 11% 9 Shuttle schedule 282 3% 54% 12% 10 Policies and procedures 240 3% 56% 14% 11 Parking, carpools/vanpools, lottery, passes 231 3% 59% 15% 12 Cafeteria menus 226 3% 61% 16% 13 Division news 221 3% 64% 18% 14 Calendars (events, meetings, holidays, pay, training) 203 2% 66% 19% 15 Retirement (Thrift Plan, gifts, parties) 201 2% 69% 20% 16 Organization charts 183 2% 71% 22% 17 Forms, memo templates 154 2% 73% 23% 18 Visitors (tours, registration) 153 2% 74% 24% Task ItB Find People poll result* Find out which immediate section/unit XX works in 1 – Employee’s immediate unit/section Find out which immediate section/unit XX works in 1 – Employee’s immediate unit/section Find out who the Chief Financial Officer reports to 3 – Place in organization chart Find out who the Director of the Corporate Affaires is 3 – Place in organization chart XXis out of the office but you need to speak to someone else on his team. Find another person who works in the same immediate section/unit 4 – Who to contact if this person is not around Find the mobile phone number for XX 5 – Employee’s mobile phone number Find an expert in Web Usability 6 – Expertise *Note: Poll result #2 was photo/picture
  12. 12. Find out which immediate section/unit XX works in (top task #1)  2010  Sharepoint 6/2013  Sharepoint 9/2013 (after changes) 80% 20% Succes s Failure Median time on task = 44 seconds 63%12% 25% Succes s Failure Median time on task = 57 seconds 100 % Succes s Failure Median time on task = 32 seconds
  13. 13. Find out which immediate section/unit XX works in (top task #1)  2010  Sharepoint 6/2013  Sharepoint 9/2013 (after changes) Median time on task = 35 seconds 100 % Succes s Failure Median time on task = 14 seconds 100 % Succes s Failure 68%6% 25% Succes s Failure Median time on task = 38 seconds
  14. 14. Find out who the Chief Financial Officer reports to (top task #3)  Old ItB (2010)*  Sharepoint 6/2013  Sharepoint 9/2013 (after changes) 40%27% Success Failure Disaster33% Median time on task = 73 seconds 38% 50% 12% Success Failure Disaster Median time on task = 135 seconds 100% Success Failure Disaster Median time on task = 120 seconds** (45 seconds) **”All sites” link causes problems
  15. 15. XX is out of the office but you need to speak to someone else on his team. Find another person who works in the same immediate section/unit (top task #4)  Old ItB (2010)*  Sharepoint 6/2013 60% 7% Success Failure Disaster 88% 12% Success Failure Disaster 33 % Median time on task = 45 seconds Median time on task = 57 seconds
  16. 16. Find out who the Director of the Office of Inspector General is (top task #3)  Sharepoint 6/2013 Median time on task = 122 seconds 38% 62% Success Failure Disaster Problems: • Searching by division acronym “OIG” doesn’t help • Mr. XXs title “Inspector General” led to confusion • Section XX org chart in Sharepoint is useless
  17. 17. Find the mobile phone number for XX (top task #5)  Sharepoint 6/2013 100 % Succes s Failure Median time on task = 14 seconds
  18. 18. Find an expert in web usability (top task #6)  Sharepoint 6/2013 63% 12% Success Failure Disaster 25% Median time on task = 52 seconds Problems: • What does “Ask Me About” really mean? Expertise? • “All sites” tab causing problems • Just not intuitive you can do this
  19. 19. Aggregate results  Old ItB (2010)  Sharepoint 6/2013  Sharepoint 9/2013 (after changes) 63% 24% Succes s Failure13% 67% 12% Succes s Failure 21% Median time on task = 52 seconds Median time on task = 76 seconds 100 % Succes s Failure Median time on task = 55 seconds** (25 seconds) **”All sites” link causes problems
  20. 20. What does this mean?  What does this mean? – People get slightly more accurate answers with Sharepoint, but it takes them longer to find those answers. To vastly improve the usefulness of Sharepoint and to make it easier to Find People, we need to improve Sharepoint.  So, how do we improve?
  21. 21. How to improve 1 – the easy stuff  Simple, high impact improvements (what worked in the September retests?) – Explicitly labeling fields (Division, Section, Job Title, etc)  Mysite pages  Search results pages – Making it easy for people to search by both acronym and full name  Divisions  Job titles – Allow search by Division (full name and acronym) from the main search box
  22. 22. How to improve 2  Other suggestions – Get rid of “All sites” button – Improve trust in the org chart viewer  Accuracy  Design (who reports to whom?) – Taxonomy must work well or people will quickly lose trust in it – Improve uniformity (job titles, dealing with misspellings, section names, division names, etc)
  23. 23. MEASURE • Find out who the Chief Financial Officer reports to (top task #3) brian@customercareowrds.com Image from: Ari Bakker http://www.sharepointconfig.com/2011/05/how-to-create-a-simple- sharepoint-2010-people-directory/
  24. 24. How to improve 1 – the easy stuff  Simple, high impact improvements (what worked in the September retests?) – Explicitly labeling fields (Division, Section, Job Title, etc)  Mysite pages  Search results pages – Making it easy for people to search by both acronym and full name  Divisions  Job titles – Allow search by Division (full name and acronym) from the main search box

×