Pam Noreault and Tara Knapp from ACI Worldwide presented on improving content user experience (UX) through user research and validation testing. They described their methodology for selecting content to improve, testing updated content with customers, and analyzing the results. Their testing showed that the updated content was located faster and tasks were completed with more success. They emphasized the importance of user research and content validation, and suggested additional ways to get customer input like social media monitoring and customer focus groups.

1. @pnoreault #LavaCon
Improve the UX of Your Content and
Prove It
Pam Noreault – ACI Worldwide
Tara Knapp – ACI Worldwide

2. Contact us
Tara Knapp
Manager, Information Development
ACI Worldwide
tara.knapp@aciworldwide.com
taraknapp@gmail.com
@tara_knapp (Twitter)
Tara Knapp (LinkedIn)
Pam Noreault
SR Manager, Information Development
ACI Worldwide
pam.noreault@aciworldwide.com
pamnoreault@gmail.com
@pnoreault (Twitter)
Pam Noreault (LinkedIn)

3. Journey
• Beginning
• Imagine - What could be
• Our road
• Content quality
• (Re)evolution
• What’s stopping you?

4. Beginnin
g

5. Imagine!
Customers are your biggest advocates.

6. Where is your content on a scale of 1 to
3?
1 = Hot Mess 2 = Partial Mess 3 = Utopia

7. Methodology before

8. Methodology now – inching upwards
Personas
(product level)
User Research and Analysis
(release level)
*usage patterns
Information Model
(release level)
User Stories
User-Centered Content
Information Model (deliverable)
Concise writing
Topic-based writing
Task-based writing
Writing for translation
Writing for accessibility
Validation Testing

9. How we select content to fix
1. Select deliverable
2. Select content to uplift

10. Before model After
models

11. Before overview
After overview

12. Before topic
After topic

13. Road led us here – BUT prove the changes made a
difference
Contextual overviews
Concise/clear content
Reduced
content/eliminated clicks
Topic-based (text
scanning)
Accessibility – checklist of
fixes
Translation – checklist of
fixes

14. How many of you are doing content validation with
customers?

15. Validation methodology
Model for PDF Documents
• Uplifted four documents
Methodology
• Teams of 4-7 writers per document
• Tested each document with at least 2 users from 2 customers
• Tested 1 hour via WebEx
• 4 tasks tested on each doc
• Presented 2 docs – old + new
• Order of docs alternated

16. Validation protocol
Directions
• Think aloud
• Tell us when you have completed the task or you give up
Test Protocol
• Each task was timed.
• Each task was completed successfully/unsuccessfully. Testers could
give up.
• After all tasks were completed on one doc, testers rated the content.
Scale: from 1 to 5.
• After all tasks were completed for both docs, testers rated their overall
experience. Scale: from 1 (poor) to 7 (great).
• Data recorded in a Google form
• Sessions recorded with permission

17. Sample task
You are a operations manager put in charge of monitoring
the system.
Task: Use the user guide to determine your two areas of
responsibility in terms of configuration.

18. ?

19. Percentage of tasks completed correctly
0%
20%
40%
60%
80%
100%
Document 1 Document 2 Document 3 Document 4
Old Doc New Doc

20. Results – Document 1 (Average before & after)
Simple 1 1.5 2 2.5 3 3.5 4 4.5 5 Complex
Effective 1 1.5 2 2.5 3 3.5 4 4.5 5 Ineffective
Clean 1 1.5 2 2.5 3 3.5 4 4.5 5 Confusing
Clear 1 1.5 2 2.5 3 3.5 4 4.5 5 Cluttered
Valuable 1 1.5 2 2.5 3 3.5 4 4.5 5 Not Valuable
Simple 1 1.5 2 2.5 3 3.5 4 4.5 5 Complex
Effective 1 1.5 2 2.5 3 3.5 4 4.5 5 Ineffective
Clean 1 1.5 2 2.5 3 3.5 4 4.5 5 Confusing
Clear 1 1.5 2 2.5 3 3.5 4 4.5 5 Cluttered
Valuable 1 1.5 2 2.5 3 3.5 4 4.5 5 Not Valuable

21. Results – Document 2 (Average before & after)
Simple 1 1.5 2 2.5 3 3.5 4 4.5 5 Complex
Effective 1 1.5 2 2.5 3 3.5 4 4.5 5 Ineffective
Clean 1 1.5 2 2.5 3 3.5 4 4.5 5 Confusing
Clear 1 1.5 2 2.5 3 3.5 4 4.5 5 Cluttered
Valuable 1 1.5 2 2.5 3 3.5 4 4.5 5 Not Valuable
Simple 1 1.5 2 2.5 3 3.5 4 4.5 5 Complex
Effective 1 1.5 2 2.5 3 3.5 4 4.5 5 Ineffective
Clean 1 1.5 2 2.5 3 3.5 4 4.5 5 Confusing
Clear 1 1.5 2 2.5 3 3.5 4 4.5 5 Cluttered
Valuable 1 1.5 2 2.5 3 3.5 4 4.5 5 Not Valuable

22. Results – Document 3 (Average before & after)
Simple 1 1.5 2 2.5 3 3.5 4 4.5 5 Complex
Effective 1 1.5 2 2.5 3 3.5 4 4.5 5 Ineffective
Clean 1 1.5 2 2.5 3 3.5 4 4.5 5 Confusing
Clear 1 1.5 2 2.5 3 3.5 4 4.5 5 Cluttered
Valuable 1 1.5 2 2.5 3 3.5 4 4.5 5 Not Valuable
Simple 1 1.5 2 2.5 3 3.5 4 4.5 5 Complex
Effective 1 1.5 2 2.5 3 3.5 4 4.5 5 Ineffective
Clean 1 1.5 2 2.5 3 3.5 4 4.5 5 Confusing
Clear 1 1.5 2 2.5 3 3.5 4 4.5 5 Cluttered
Valuable 1 1.5 2 2.5 3 3.5 4 4.5 5 Not Valuable

23. Results – Document 4 (Average before & after)
Simple 1 1.5 2 2.5 3 3.5 4 4.5 5 Complex
Effective 1 1.5 2 2.5 3 3.5 4 4.5 5 Ineffective
Clean 1 1.5 2 2.5 3 3.5 4 4.5 5 Confusing
Clear 1 1.5 2 2.5 3 3.5 4 4.5 5 Cluttered
Valuable 1 1.5 2 2.5 3 3.5 4 4.5 5 Not Valuable
Simple 1 1.5 2 2.5 3 3.5 4 4.5 5 Complex
Effective 1 1.5 2 2.5 3 3.5 4 4.5 5 Ineffective
Clean 1 1.5 2 2.5 3 3.5 4 4.5 5 Confusing
Clear 1 1.5 2 2.5 3 3.5 4 4.5 5 Cluttered
Valuable 1 1.5 2 2.5 3 3.5 4 4.5 5 Not Valuable

24. Average overall ranking – 1 poor to 7 great
0
1
2
3
4
5
6
7
Document 1 Document 2 Document 3 Document 4
Old Doc New Doc

25. • Content located faster in 3 of 4 new models.
• Validation tasks completed with increased
success in 2 of 4 new models.
• Content rating higher in 3 of 4 new models.
• Overall content ranking higher in 2 of 4 new
models.

26. Laugh and cry moments
• Surfing & browsing
• Clueless & perfectionist
• Change haters
• Fear of failure
• When is done really done
• Aha moments
• Technical snafus
• Testing heavy-duty reference
content was a bust

27. Customers are their biggest advocates.
What the writers learned

28. Lessons the writers came up with
• Do a dry run
• Observing the users’ choices can be as
useful as the data
• Cannot predict how users will do the tasks
• Define what “done” means
• Reference-based content should not be
tested with the same methodology as task-
based content
• Rebooting your computer prior to testing
has its benefits
• Repeated contact with customers removes
the fear factor over time

29. 1. We require user research and content validation, where
appropriate.
2. We get creative…..
• Collaborate with people who work with customers
• Monitor and mine data from social networking sites
• Join LinkedIn groups - ask questions & post surveys
• Seek input from people who represent the same personas as our users
• Participate in customer-focus groups (Design Partner Programs)
Re(evolution)

30. What are you waiting for?

31. Pam Noreault
• pam.noreault@aciworldwide.com
• pamnoreault@gmail.com
• @pnoreault (Twitter)
• Pam Noreault (LinkedIn)
Tara Knapp
• tara.knapp@aciworldwide.com
• taraknapp@gmail.com
• @tara_knapp (Twitter)
• Tara Knapp (LinkedIn)