The document summarizes the findings of a competitive UX benchmarking study of four major healthcare insurance websites: Blue Cross Blue Shield sites in Texas, New Jersey, California, and Massachusetts. The methodology involved both quantitative usability testing with 200 participants and qualitative testing with 20 think-out-loud participants across the four sites. Key performance indicators were measured, including task success rates, time on task, problems encountered, and attitudinal ratings. Overall, the Texas BCBS site scored highest across metrics, while the Massachusetts site saw the biggest drop in brand perception after usability testing. Common problems included difficulty comparing plans and understanding terminology.

1. Competitive UX Benchmarking: How
Four Healthcare Insurance Sites
Scored Across the Board
Kuldeep Kelkar | Dana Bishop | April 12, 2018

2. Quick Housekeeping
• Control panel on the side of your screen if you
have any comments during the presentation
• Time at the end for Q&A
• Today’s webinar will be recorded for future
viewing
• All attendees will receive a copy of the
slides/recording
• Continue the discussion using #uzwebinar
Let’s make sure you’re all set up for the webinar!

3. Agenda
UX Benchmarking
Kuldeep Kelkar
Methodology
KPI, qxScore on uzIndex
Key Findings & Insights
Tips & Tricks | Resources
Q & A
Dana Bishop

4. UX Benchmarking
LONGITUDINAL
BENCHMARKING
COMPETITIVE
TAKE
ACTION
COLLECT
DATA
ANALYSIS

5. Longitudinal UX Benchmark
81.1
77.3
82.3
76.6
77.2
81.2
40
50
60
70
80
90
100
Oct-16 Nov Dec Jan Feb Mar Apr May June July Aug Sept Oct-17
Task 1 Task 2 Task 3
Differences of approximately +/- 12% is significant at the 90% confidence
level
Significant
improvement

6. Competitive UX Benchmark
ü How your site performs relative to your competitors
ü Interpret your usability standing, feature set and more within your industry
ü Learning from their success and failures, what works or does not for your competitors
ü Industry best-in-class examples to emulate
ü Sometimes benchmark with other Industries

7. Poll
How many UX benchmarks have you conducted In the last 12 months?
• None
• None, but thinking about or planning to conduct one soon
• One benchmark
• More than one

8. Poll
How many competitors (in addition to your own site), would you typically want to benchmark against?
• 1
• 2 or 3
• 4 or 5
• 5+
• Not sure at this time

9. Designing a benchmarking study
TAKE YOUR TIME.
DESIGN ONCE, RUN MULTIPLE TIMES.
Tasks should be consistent across all
competitors. Review each site to
confirm all tasks can be completed
across competitors.
CONSISTENCY
Carefully consider the usability
and business metrics you want
to track.
KEY PERFORMANCE INDICATORS (KPIs)
Makes it quick and easy to
design and build your
benchmark, then repeat/ re-
run as needed in the future.
USERZOOM PLATFORM
Include the primary tasks users
do on your site or app.
A Retail benchmark might include:
1. Browse for Product
2. Search for Product
3. Purchase Process
TASKS

10. Healthcare Competitive UX Benchmarking
Methodology

11. 4 Blue Cross Blue Shield (BCBS) websites were tested.
TEXAS
NEWJERSEYMASSACHUSETTS
CALIFORNIA

12. Two type of studies
Quantitative N = 200
with Screen Recording
1
2 Qualitative N = 20
With Think Out Loud

13. Behavior and Attitudinal Data was collected
Actual Behavior is recorded and observed Attitudinal ratings are captured pre & post task
Study Participants go
through tasks in their
natural environment at
their own pace, using
UserZoom platform

14. Think out loud (for Study 2, N = 20)
Study Participants
verbalize Talk-Out
Loud while going
through the task

15. Between Subject Experiment
Initial
Questionnaire
Final
Questionnaire
n=50 n=50 n=50 n=50
Task 1:
Compare and Select Plans
Task 1:
Compare and Select Plans
Task 2:
Find Copay
Task 3:
Find a Specialist
Task 2:
Find Copay
Task 3:
Find a Specialist
Task 1:
Compare and Select Plans
Task 2:
Find Copay
Task 3
Find a Specialist
Task 1:
Compare and Select Plans
Task 2:
Find Copay
Task 3:
Find a Specialist
Texas New Jersey California Massachusetts

16. The “right” participants
Reside in the state | 26+ | Responsible for making healthcare decisions

17. Key Performance Indicators (KPIs)

18. KPI Model
Level 1
Level 2
Level 3
Scale & Activity KPIs: How many? How much?
Value & Utility:
Are they using it?
Financial KPIs
User Experience:
Can they use it?

19. KPI Model
Level 1
Level 2
Level 3
Scale & Activity KPIs: How many? How much?
Acquisition Conversion | Social
Sharing | Downloads
Time on Site | Churn Rate
Financial KPIs
Journey Success | Problems &
Frustrations | Usability | Trust |
Appearance | Emotional Benefits
Value & Utility: Are they using it? User Experience: Can they use it?
Monthly active users | Activity Frequency | # of Transactions
Transaction Volume | Churn Volume | Abandonment Rates
Revenue | Margin | Cash Flow | Net Profits

20. KPI Model
Level 1
Level 2
Level 3
Scale & Activity KPIs: How many? How much?
Acquisition Conversion | Social
Sharing | Downloads
Time on Site | Churn Rate
Financial KPIs
Journey Success | Problems &
Frustrations | Usability | Trust |
Appearance | Emotional Benefits
Value & Utility: Are they using it? User Experience: Can they use it?
Monthly active users | Activity Frequency | # of Transactions
Transaction Volume | Churn Volume | Abandonment Rates
Revenue | Margin | Cash Flow | Net Profits

21. UX KPIs
BEHAVIORAL (WHAT THEY DID) ATTITUDINAL (WHAT THEY SAY)
Ease of Use
Trust & Credibility
Appearance
Loyalty, NPS
Task Success
Task Time
Page Views, Clicks
Problems & Frustrations

22. qxScore
qxScore
Overall Task Success %
Trust
Ease of Use Appearance
Loyalty NPS
Task 1 Success %
Task 2 Success %
Task N Success %
Behavior: What they do Attitude: What they say, feel

23. uzIndex
0%
0%
0%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Website 4
Website 3
Website 2
Website 1
GreatGoodAveragePoorVery Poor

24. Key Findings and Insights

25. Blue Cross Blue Shields (BCBS) Texas was the overall winner!
48%
56%
63%
76%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Massachusetts
New Jersey
California
Texas
GreatGoodAveragePoorVery Poor

26. Pre-Post Brand Perception
BCBS MA saw the biggest drop in brand perception after 3 tasks on the site
68%
64% 62%
75%
70%
46% 46%
40%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Texas New Jersey California Massachusetts
Pre Pre Pre PrePost Post Post Post
Selected 6-7 on 7-point scale where 1 = Negative and 7 = Positive
What is your perception of [SITE]? Top 2 Box on 7 point scale

27. Task 1: Compare and Select Plan
Assume that you lost your job and need to find healthcare insurance for you and your
spouse. You’re both generally healthy but do get sick a few times a year and will occasionally
visit specialists as needed.
Compare healthcare plans and select one that will meet your needs.
Your birthday: 11/11/1984 (Age 31)
Spouses birthday: 12/12/1986 (Age 29)
Household income: $150,000
Zip Code: 78703

28. Task 1: Compare and Select Plan
KPIs
TX
n=50
NJ
n=50
CA
n=50
MA
n=50
Task Success 94% 56% 88% 64%
Average Time on Task 3.3 min 3.8 min 5.8 min 4.2 min
Average # of Page Views 4.9 pages 6.7 pages 9.0 pages 9.0 pages
Did not encounter problems or frustrations 38% 24% 22% 20%
Provided just the right amount of information 58% 46% 48% 44%

29. Task 1: Even the winner TX had several problems & frustrations
28%
24%
20%
20%
18%
14%
10%
6%
6%
6%
2%
6%
38%
Unsure which plan is best for me
Difficult to find plan details
Difficult to compare plans
Difficult to understand plan differences
Unsure which type of plan to choose
Terminology was confusing / difficult to understand
Didn’t understand why personal info was required to find a plan
Text was too small / difficult to read
I was taken to another site
Didn’t trust the site with my information
Difficult to understand the meaning of acronyms used
Other, please specify
I did not encounter any problems or frustrations
TX N=50
More specifically, which of the following problems or frustrations, if
any, did you encounter during this task? [Check all that apply]

30. Task 1: MA site had similar issues, but in greater numbers
40%
36%
34%
28%
20%
18%
18%
14%
8%
8%
4%
10%
20%
Difficult to compare plans
Unsure which type of plan to choose
Unsure which plan is best for me
Difficult to understand plan differences
Didn’t understand why personal info was required to find a plan
Difficult to find plan details
Terminology was confusing / difficult to understand
Difficult to understand the meaning of acronyms used
Text was too small / difficult to read
I was taken to another site
Didn’t trust the site with my information
Other, please specify
I did not encounter any problems or frustrations
MA N=50
More specifically, which of the following problems or frustrations, if
any, did you encounter during this task? [Check all that apply]

31. Task 1: Problem Video Example for MA site
Participant ID 67:
“I’m not sure what all these things are. There’s a lot of things packed into a small space”

32. Task 1: NJ Challenges (Personal Information)
Compared to the other sites, the New Jersey Horizon Blue site required more
personal information to find plan information.
20% didn’t understand why so much personal information was required.
“The process was too lengthy”
“…the compare plans” page gives NO
information about what the plans
actually entail. In order to have
more comprehensive service, I had
to fill out an actual application
including email address and
personal data…”
“Entering information can be overwhelming. I
wish the process in getting a quote walked me
through the information was a bit simpler”
“No way to get other information
without entering a lot of it first”

33. Task 2: Find Copay
Now that you’ve selected a plan, determine the following:
Find the copay amount for a primary care office visit. Write down the copay amount.

34. Task 2 Find Copay KPIs
Participants picked up from where they left off on the previous task; those who were not successful in finding a plan
tended to struggle in determining the copay. Texas site had highest Task Success.
KPIs
TX
n=50
NJ
n=50
CA
n=50
MA
n=50
Task Success 94% 64% 82% 60%
Average Time on Task 0.7 min 0.9 min 1.4 min 0.8 min
Average # of Page Views 1.5 pages 2.3 pages 2.1 pages 2.8 pages
Ease of Finding Copay Information 66% 44% 48% 50%
Did Not Encounter Problems or Frustrations 64% 48% 42% 52%

35. Task 3: Find a specialist
Pretend that you’re interested in a PPO Plan. Find a list of specialists you can see for allergies
in the [AREA] who is accepting new patients. You would prefer a female doctor who speaks
Spanish.
Review the information provided about the doctors. Keep in mind the following:
1) Does the site provide the type of information you need when searching for a doctor?
2) What other types of information would you like to see?

36. Task 3: Find Specialist KPIs
Texas had the highest overall rate of task success for finding a specialist, however other KPIs were
higher on the New Jersey site
KPIs
TX
n=50
NJ
n=50
CA
n=50
MA
n=50
Task Success 70% 58% 52% 40%
Average Time on Task 2.8 min 1.9 min 3.3 min 3.8 min
Average # of Page Views 4.7 pages 4.3 pages 5.1 pages 6.1 pages
Ease of Task Completion 40% 52% 34% 8%
Did Not Encounter Problems or Frustrations 40% 48% 28% 8%

37. Task 3 - Find Specialist: Problems and Frustrations - MA
Nearly half of MA participants struggled with searching by specialty and almost a third were
unsure where to go to find a doctor
48%
30%
14%
12%
10%
10%
10%
6%
4%
4%
2%
10%
8%
Couldn’t search by specialty
Unsure where to go to find a doctor
I was taken to another site
Difficult to narrow search by gender
Not enough information provided about the
No doctors available in the area
Difficult to narrow search by language
Unclear if the doctor is accepting new patients
Terminology was confusing / difficult to
Difficult to understand the meaning of acronyms
Text was too small / difficult to read
Other, please specify
I did not encounter any problems or frustrations
MA N=50
More specifically, which of the following problems or frustrations, if
any, did you encounter during this task? [Check all that apply]

38. Task 3 Find Specialist: Frustrations with TX site
Even the highest task 3 success site (TX) had frustration. Lets look at one example.
28%
12%
10%
8%
8%
4%
4%
4%
4%
2%
2%
14%
40%
Difficult to narrow search by gender
Couldn’t search by specialty
Unsure where to go to find a doctor
Not enough information provided about the
No doctors available in the area
I was taken to another site
Difficult to narrow search by language
Unclear if the doctor is accepting new patients
Text was too small / difficult to read
Terminology was confusing / difficult to
Difficult to understand the meaning of acronyms
Other, please specify
I did not encounter any problems or frustrations
TX N=50
More specifically, which of the following problems or frustrations, if
any, did you encounter during this task? [Check all that apply]

39. Task 3: Texas (TX) Challenges to narrow by gender
Search and filter criteria were on two different pages
28% didn’t see the ability to filter by gender on the
search results page
“Couldn’t find a way to specify male vs.
female doctor – had to go through the
list” – ID 3096
“There was no option to choose
a female doctor…” – ID 2074

40. Task 3 Find Specialist: Frustrations with NJ site
22% of NJ participants could not find a specialist meeting the criteria specified due to problems with search
22%
14%
10%
8%
6%
6%
6%
2%
2%
2%
0%
10%
48%
No doctors available in the area
Unsure where to go to find a doctor
Difficult to narrow search by language
I was taken to another site
Difficult to narrow search by gender
Not enough information provided about the doctors
Unclear if the doctor is accepting new patients
Couldn’t search by specialty
Difficult to understand the meaning of acronyms
Text was too small / difficult to read
Terminology was confusing / difficult to understand
Other, please specify
I did not encounter any problems or frustrations
NJ N=50
More specifically, which of the following problems or frustrations, if
any, did you encounter during this task? [Check all that apply]

41. Task 3 New Jersey (NJ) site frustration: No doctors in the area
If participants left the
“Accepting new patients”
box unchecked àresults
were returned and upon
clicking on individual
doctors they could find
doctors meeting the criteria
provided who are accepting
new patients.
May be a bug?
22% struggled with finding a specialist meeting the criteria specified in the area. If participants checked the “Accepting new
patients” box prior to search, they saw 0 results in the Newark area, even upon expanding the radius up to 50 miles

42. To summarize

43. To Summarize
ü Quantitative Details: qxScore on uzIndex, Task
Success, Problems and Frustrations
ü Qualitative Details: Participant Videos,
Screenshots & Heat maps
Hearts and Mind

44. Don’t wait… Get Started now
DESIGN ONCE, RUN MULTIPLE TIMES.
Tasks should be consistent across all
competitors. Review each site to
confirm all tasks can be completed
across competitors.
CONSISTENCY
Carefully consider the usability
and business metrics you want
to track.
KEY PERFORMANCE INDICATORS (KPIs)
Makes it quick and easy to
design and build your
benchmark, then repeat/ re-
run as needed in the future.
USERZOOM PLATFORM
Include the primary tasks users
do on your site or app.
A Retail benchmark might include:
1. Browse for Product
2. Search for Product
3. Purchase Process
TASKS

45. Related Webinars On-Demand
Visit Userzoom.com > resources > webinars
Sourcing The
Right Participants
For Your UX
Research & Testing
Moving The
Needle With UX
Benchmarking
userzoom.com/webinar/sourcing-right-
participants-ux-research-testing/
userzoom.com/webinar/moving-the-needle-with-
ux-benchmarking/
1 2

46. Q&A

47. THANK YOU
marketing@userzoom.com +1 866-599-1550