1.
How to measure
content effectiveness
@meghscase
@willfredsiem
#ConfabINT

2.
Just a little big longer …
• Overview of measurement methods
• Our client for the day
• Measurement frameworks
• User testing practice
• Content scorecards

3.
Overview of Methods

4.
Three-pronged approach
Analytics
Heuristics
User Testing

5.
Analytics
Can answer questions like:
• What pages are viewed the most?
• What do people search for on the site?
• Are our users more likely to use navigation or search to find
information?
• What paths do people take through our site?
• Where do people exit the site?
• How long do people spend on the site or a page?
• Do people click on our calls to action?
• How do visitors get to our site?
• What content on a page do people look at?

6.
Heuristics
Can answer questions like:
• Does the content communicate our key
messages?
• Is the content relevant to our strategy?
• Is the content easy to scan and read?
• Does the content contain clear calls to action or
next steps?
• Does the content contain lingo,
jargon, or fluff?
• Is the content easy-to-find
through search and navigation?
• Does the content conform to
accessibility guidelines?

7.
User Testing
Can answer questions like:
• Is our content organized intuitively?
• Is the content easy-to-understand?
• Do users know what the next steps are after
reading the content?
• Is the content relevant to users’ needs?
• Does the content make users feel confident and
informed?

8.
Our Client for the Day

9.
U.S. Department of State

10.
Client Brief
•GOAL: Reduce call center costs
•RATIONALE: 75% of calls to the call center are on topics
that web content should be able to address.
•USE CASES: The primary use cases for those calls are:
- Business traveler needing to travel in 2-4 weeks
- Person needing to travel for a family emergency
- Parent who needs to get a passport for their child

11.
Content Strategy Statement
The passport section on travel.state.gov helps us contain
call center costs with easy-to-understand, actionable content
for U.S. business, leisure, and emergency travelers that
explains how to apply for a passport and makes it easy to
apply successfully.

12.
Content Strategy Statement
The passport section on travel.state.gov helps us contain
call center costs with easy-to-understand, actionable
content for U.S. business, leisure, and emergency travelers
that explains how to apply for a passport and makes it easy
to apply successfully.

13.
Content Strategy Statement
The passport section on travel.state.gov helps us contain
call center costs with easy-to-understand, actionable
content for U.S. business, leisure, and emergency travelers
that explains how to apply for a passport and makes it
easy to apply successfully.

14.
Content Strategy Statement
The passport section on travel.state.gov helps us contain
call center costs with easy-to-understand, actionable content
for U.S. business, leisure, and emergency travelers that
explains how to apply for a passport and makes it easy to
apply successfully.

15.
Measurement
Frameworks

16.
Definitions
•KPIs: How an organization determines, with data,
whether it’s meeting its goals.
•Objective: How a tactic should influence a KPI.
•Metric: How you determine whether an objective
influenced a KPI.

17.
Measurement Framework

18.
Activity
1. Grab an easel for your table.
2. Make a content measurement
framework worksheet.
3. Work together to fill out the
worksheet based on our client
brief for today. Use post its, or just
write on the worksheet.
KPI Content Objective
Metric Method

19.
User Testing Practice

20.
Content-Focused User Tests
• Cloze test
• User questions test
• Highlighting test

21.
User Test 1
Type: Cloze Test
Tests for: Ease of comprehension
Mode: Digital or print
Execution: In-person or remote

22.
Call us to make an appointment for emergency passport
service in case of a life or death emergency.
Life or death emergencies are serious illnesses, injuries, or
deaths in your immediate family that require you to travel
outside the United States within 72 hours (3 business days).
You must provide all the usual application documentation in
addition to proof of the life or death emergency.

23.
Call us to make an appointment for emergency passport
service in case of a life or death emergency.
Life or death emergencies are serious illnesses, injuries, or
deaths in your immediate family that require you to travel
outside the United States within 72 hours (3 business days).
You must provide all the usual application documentation in
addition to proof of the life or death emergency.

24.
Discuss
• Talk about any difficulties figuring out the
missing words and how meaning was
changed if you chose a different word
• Come up with one or more
recommendations for how the content
could be more clear

25.
User Test 2
Type: User questions test
Tests for: Content relevance
Mode: Digital or print
Execution: In-person or remote

26.
Practice
Get into groups of two at your tables.
• You will take turns being the administrator and the user.
• The administrator will ask the user to jot down the questions they
have when presented with a title of a web page.
• Once the user has jotted down their questions, administrator will ask
them to read the web page.
• After the user has read the page, the administrator will ask them what
their questions were and whether their questions were answered. The
administrator can probe further with follow up questions.
• After the discussion, the administrator should write down one idea for
improving the content.

27.
Group 1
Page title:
Get a passport in a hurry
URL:
http://tiny.cc/test1group1
1. Administrator asks user to jot
down questions they’d have
for a page with this title.
2. After user jots down questions,
the administrator shows the
user the page and asks them
to read it.
3. After the user reads the page,
administrator facilitates a
discussion.
4. Administrator writes down
one recommendation for
improving the content.

28.
Group 2
Page title:
Passport photos
URL:
http://tiny.cc/test1group2
1. Administrator asks user to jot
down questions they’d have
for a page with this title.
2. After user jots down questions,
the administrator shows the
user the page and asks them
to read it.
3. After the user reads the page,
administrator facilitates a
discussion.
4. Administrator writes down
one recommendation for
improving the content.

29.
Discuss
• How’d it go as a user and
administrator?
• What did you learn?
• What recommendations did you
have for improving the content?

30.
User Test 3
Type: Highlighting test
Tests for: Content effectiveness, voice and tone
Mode: Digital or print
Execution: In-person or remote

31.
Practice
Stick with your partner or switch it up if you like:
• You will take turns being the administrator and the user.
• The administrator will open a content document on a computer, phone, or tablet. A
computer or tablet will work best.
• The administrator will explain to the user that they’d like to understand how the
content makes them feel, e.g., at-ease or worried, informed or confused. They will
ask the user to highlight in a reddish color the content in the document that made
them feel the negative emotion/state and highlight in a greenish color the content
that made them feel the positive emotion/state.
• The user will complete the test.
• Afterwards, the administrator will review how they highlighted and ask follow-up
questions, such as “Why did that sentence make you feel worried?”
• Finally, the administrator will write down one or two ideas for improving the content.

32.
Group 1
Highlighter Details:
Red: Worried
Green: At-ease
URL:
http://tiny.cc/test2group1
1. The administrator will open a content
document on a computer, phone, or tablet, and
either download the document or create a
copy. A computer or tablet will work best.
2. The administrator will explain to the user that
they’d like to understand how the content
makes them feel, e.g., confident or worried,
informed or confused. They will ask the user to
highlight in a reddish color the content in the
document that made them feel the worried
highlight in a greenish color the content that
made them feel at-ease.
3. The user will complete the test.
4. Afterwards, the administrator will review how
they highlighted and ask follow-up questions,
such as “Why did that sentence make you feel
worried?”
5. Finally, the administrator will write down one or
two ideas for improving the content.

33.
Group 2
Highlighter Details:
Red: Confused
Green: Informed
URL:
http://tiny.cc/test2group2
1. The administrator will open a content
document on a computer, phone, or tablet, and
either download the document or create a
copy. A computer or tablet will work best.
2. The administrator will explain to the user that
they’d like to understand how the content
makes them feel, e.g., confident or worried,
informed or confused. They will ask the user to
highlight in a reddish color the content in the
document that made them feel confused and
highlight in a greenish color the content that
made them feel informed.
3. The user will complete the test.
4. Afterwards, the administrator will review how
they highlighted and ask follow-up questions,
such as “Why did that sentence make you feel
worried?”
5. Finally, the administrator will write down one or
two ideas for improving the content.

34.
Discuss
• How’d it go as a user and
administrator?
• What did you learn?
• What recommendations did you
have for improving the content?

35.
Content Scorecard

36.
Why?
• Put data and insights in context.
• Communicate content.
challenges and opportunities.
• Use data and insights to make
improvements.

37.
Example Scorecard

38.
KPI
Reduce calls to call center
on key self-service topics.
Content Objective
Provide clear, relevant, actionable content for the three most common calls
that should be self-service:
- Passports for life and death situations
- Passports for children under 16
- Passports for trips in 2-4 weeks
Metric
Content is easy-to-
understand.
Content gives users
confidence to act.
Content answers the key
questions users have
about the topic.
Assessment Recommendations
Activity

39.
Wrap Up

40.
What next?
• Don’t have a measurement framework? Pull together the right people
and make one.
• Have a measurement framework? Review what you measure and
how you use the information and adjust if needed.
• Practice administering the methods we learned today and try them
out with actual users.