Success, Failure, Disaster: Cisco Measures Top Tasks - CapCHI 21 Oct 2015

Success, Failure,
Disaster:
Cisco Measures Top Tasks
CapCHI Presentation by Mike Atyeo, Neo InsightOctober 20th
, 2015

2© Customer Carewords Ltd. customercarewords.com
Chat – 10:00am No 139 in queue for Microsoft Answer Tech
2
10:45 Chat starts

3© Customer Carewords Ltd. customercarewords.com 3

Out-of-control web
publishing
8

• Business goals are not clear
• Role of web is not clear
• Business goals are not tied into day-to-day design
• Imbalance of power between organizational entities
• ‘Path of least resistance’ is organization-centric
• Difficult to work across organizational boundaries
• Don’t know what users’ (top) tasks and needs are
• Don’t have user research skills
• Don’t use research skills as part of design process
• Reward production, rather than outcome
• HIPPOs drive design
• Complexity not managed
• Have no appropriate measures of outcomes
• Measuring the wrong things

Bill Skeet
Senior Manager
Digital Service and Support UX
Cisco Systems
@Cisco
Mike Atyeo
Co-founder and President
Neo Insight
@neoinsight
Gerry McGovern
Founder and CEO
Customer Carewords
@gerrymcgovern

Employees
1995:
<4,000
2015:
70,000
1995 - 2015

Temkin Group 2013 IT Decision-Maker Survey
=
More loyal, more committed
customers who will buy more
and stay longer
‘Easy to do business with’

Triangulate to the truth
Best
Answer
Customer Satisfaction
surveys (mthly/qtrly)
Customer Satisfaction
surveys (mthly/qtrly)
Site Statistics
(mthly/qrtly)
Site Statistics
(mthly/qrtly)
Task Studies
(semi-annual)
Task Studies
(semi-annual)
•Feedback links
(unsolicited)
•Customer interviews
•Social Media
•Metrics/Analytics reports
•Path analysis
•Trending
•Multivariate testing
•Usability testing
•Ethnographic research
•Participatory studies
14

Cisco customers: channel preference: opportunity for
improvement
First
channel
choice
Phone
18% of contact
volume
Phone
18% of contact
volume
Website
44% of contact
volume
Website
44% of contact
volume
Web chat
3% of contact
volume
Web chat
3% of contact
volume
Email
20% of contact
volume
Email
20% of contact
volume
Community
8% of contact
volume
Community
8% of contact
volume
First
channel
outcome
Second
channel
15

Tasks
The ‘long neck’: Users’ priorities
Top Tasks
Tiny Tasks
“Love them.”“Love them.”
“Leave
them.”
“Leave
them.”

Tasks
The ‘long neck’: Management focus
How much relative
resource to apply
Which user tasks
to focus on.
“Strategy renders choices about what not to do as
important as choices about what to do... Some
managers mistake ‘customer focus’ to mean they
must serve all customer needs or respond to every
request...”
Michael Porter

Current
website
Customers
Stakeholders
Competitors,
strategies and
other documents
Tasks:
Long list Short list Survey to uncover
the customers’ top
tasks
Top Task Identification process—
change management role
Top Task Identification process—
change management role
New website
Understanding,
buy-in and
commitment
Stakeholders

Cisco Results – 2014
1.Download software, firmware, drivers,
patches, updates
Configuration, set up, deployment
Pricing
Specifications / data sheets
Contact support / support requests
(open, check status, escalate)
The Top task has as
many votes as the
bottom 26 tasks

Continuous Improvement

Scheduled in advance 45m – 1 hour Scripted, pre-defined
tasks, questions

Anatomy of a task
“Your 7600 router crashed after entering the “show mpls
forwarding with owner …” command. Determine if there
are any workarounds for this known issue.”
Sample task: Find Bug Information
TPI tasks are specific
versions of the most
important, typical tasks.
In a Task Performance
Indicator study we ask 18-
20 subjects to perform 10-
12 tasks in 1 hour.
TPI testing is performed
remotely, on the live site.
No ‘think-aloud’ protocol

Task Success
Rates
Other measures
contributing to TPI
scores
•Completion rates
• Success, Failure, ‘Disaster’
• Failure codes
• Time on Task
• Confidence
• In success
• In failure
• Event codes
• Specific pages visited
• Usability issues encountered

Turning Results into Action
Top Tasks
Management diagnoses
what is wrong and
clearly indicates how
to fix it.
JUNE 2012 Bug Task - 18 Subjects
Jun 2012
TPI 23TPI 23
5 Succeed5 Succeed
6 Give up6 Give up
6 Time out or
partial
answer
6 Time out or
partial
answer
1 Disaster1 Disaster

Memorable Task Performance Indicator scores,
setting targets
TPI scores are
between 0 and 100
Original Score = 23
70
FY15
Task Performance
Target =
28

Video: One participant finds most of the
problems!

Bug Task Success Rate Comparisons 2012 - present
Dec 2014
TPI 49TPI 49
Nov 2013
TPI 6TPI 6
Apr 2014
TPI 3TPI 3
Jul 2014
TPI 25TPI 25
Turning Results into Action
Monitor Top Task performance on
an ongoing basis
Oct 2012 May 2013 July 2013Feb 2012 Jun 2012 Feb 2013
TPI 6TPI 6TPI 15TPI 15 TPI 17TPI 17 TPI 39TPI 39TPI 23TPI 23TPI 8TPI 8
2015 TPI Goal: 55
Ran a pilot test to simulate
a possible solution. It
worked!
Ran a pilot test to simulate
a possible solution. It
worked!
Sample task: Find Bug Information

Minor Issues (50-59) TPI = 50
Task: Register (Create new account)
‘Novice’ participants only

‘Register’ task performance over 8 rounds of
testing
32
Nov 2012Nov 2012 May 2013May 2013 Aug 2013Aug 2013 Dec 2013Dec 2013
Primarily resolving bugs
and technical issues
Primarily resolving bugs
and technical issues
(Novices only) Create new account: Create a new guest account to access the
Cisco.com website and log in with this new account.
Mar 2014Mar 2014

Jun 2015Dec 2014
1. Make it clear which fields are
mandatory
2. Have clearer password
strength indicator (or
requirements)
3. Fix ‘inconsistent address
combination’ errors
4. Reduce the number of form
fields
22 mandatory fields
and selections
12 mandatory fields
and selections
Shorter form has cumulative impact:
•Participants are less likely to skip fields – they
don’t even THINK about which fields are
mandatory!
•Fewer opportunities for poor design / for users
to make mistakes
•Shorter time on form, so less likely to hit 5-
minute Time Limit completing the task
And – probably:
•People are more likely to Register
•Fewer calls to help desk
•Fewer abandonments

‘Register’ task performance over 8 rounds of
testing
34
(Novices only) Create new account: Create a new guest account to access the
Cisco.com website and log in with this new account.
Mar 2014Mar 2014 Aug 2014Aug 2014 Jan 2015Jan 2015 Jul 2015Jul 2015
22 fields + NO design
improvements
12 fields + 3 design
improvements
22 fields + 3 design
improvements
12 fields+ 3 design
improvements

Building the business case: Registration task
From Q1 FY 2014 to Q3 FY 2015:
•Registration completion has more than doubled
•Number of support tickets for Registration less than half
•Time to Register reduced by one-third
35

“The TPI allows you to be a manager.” –
Bill Skeet
36
• We have a model of the contribution of the web – as
one of many channels – to the business bottom line.
• We have priorities from the customer perspective –
their top tasks.
• We have performance benchmarks.
• We have data on the performance and potential for
improvement – where the biggest bang for the buck
is.
• We can prioritize our actions and reliably calculate
what the score will be when problems are fixed.
• Thus, we can set annual goals. Goals that are
empirical, measurable and that accurately reflect the
user experience of the site.
“You can prove you’re managing the
website, not the other way around.”

Mike Atyeo
Neo Insight
Thank You
mike@neoinsight.com
@neoinsight
(613) 599-7470

Resources and References
• Customer Carewords
– Webinars: http://www.customercarewords.com/category/previous-
webinars/
– Improving Cisco Support Website Using Top Tasks Management
• http://www.customercarewords.com/improving-cisco-support-website-
using-top-tasks-management/
• Gerry McGovern’s YouTube channel
– https://www.youtube.com/user/Velvetu1000
– Overview of Top Task Identification process (15 min video)
• https://www.youtube.com/watch?v=yYUN4awIaLM
– Improving software downloads at Cisco (5-minute video)
• https://www.youtube.com/watch?v=E4plAa_aC3A&feature=youtu.be
– Improving customer log-in at Cisco (5-minute video)
• https://www.youtube.com/watch?v=1Uz0vkd3RRE
38

Success, Failure, Disaster: Cisco Measures Top Tasks - CapCHI 21 Oct 2015

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Success, Failure, Disaster: Cisco Measures Top Tasks - CapCHI 21 Oct 2015

Similar to Success, Failure, Disaster: Cisco Measures Top Tasks - CapCHI 21 Oct 2015 (20)

More from Neo Insight

More from Neo Insight (6)

Recently uploaded

Recently uploaded (20)

Success, Failure, Disaster: Cisco Measures Top Tasks - CapCHI 21 Oct 2015