In 2006, I conducted a usability evaluation of printed credit card disclosure materials for the U.S. Government Accountability Office. This presentation covers the three methods used -- readability analysis, expert review using plain language guidelines and usability test -- to assess usability. I gave the presentation at the Plain Language Association International Conference in Sydney, Australia, in October of 2009.

1. Using multiple methods
to assess the usability of
credit card disclosures
Angela Colter
acolter@gmail.com

2. What are disclosures?
card member
agreements
solicitation
letters
Using Multiple Methods | Angela Colter

3. Why was the U.S. Congress interested?
1. Assess the need to revise disclosures so they better
inform consumers of now-common penalty rates and
fees
2. Find out how effectively pricing practices are disclosed
to cardholders
3. Determine whether penalty charges contribute to
cardholder bankruptcies
Using Multiple Methods | Angela Colter

4. Three evaluation methods
1. Readability analysis using formulas
2. Expert review using plain language guidelines
3. Usability test
Using Multiple Methods | Angela Colter

5. Why those methods?
Strengths Weaknesses
Readability Quantitative data Only counts what can be
formulas counted
Quick & cheap
Doesn’t tell you what to do
about it
Plain Predicts problems and gives Doesn’t mean users will
language guidelines for improvement have problems
review Also relatively cheap
Usability test Shows how people actually Not quick and not cheap
use an interface
Small sample size can be a
Provides quantitative and hard sell
qualitative data
Using Multiple Methods | Angela Colter

6. Readability analysis
• Used FOG, SMOG, and Flesch Grade Level
• Took multiple samples
• Carefully prepped the text
(removed URLs, trailing periods and hard returns)
• Used Readability Analyzer software
• Explained strengths and weaknesses of readability
formulas in the final report
Using Multiple Methods | Angela Colter

7. Readability analysis findings
Average grade level
12 of cardmember
agreements
Flesch Grade Level
SMOG
FOG
Using Multiple Methods | Angela Colter

8. Readability analysis findings
Average grade level
17 of annual percentage
rate info
Average grade level
12 of cardmember
agreements
Flesch Grade Level
SMOG
FOG
Using Multiple Methods | Angela Colter

9. Readability analysis findings
Average grade level
17 of annual percentage
rate info
Average grade level
12 of cardmember
agreements
8 Median reading level
of U.S. adults
Flesch Grade Level
SMOG
FOG
Using Multiple Methods | Angela Colter

10. Expert review
Used a plain language
guidelines document
developed by the U.S.
Securities and Exchange
Commission in 1998
Deals with organization,
content and design http://www.sec.gov/news/extra/
handbook.htm
Using Multiple Methods | Angela Colter

11. Expert review findings
The disclosures violated many of the guidelines.
The report listed each guideline and gave examples
where disclosures followed or violated them.
Guideline: Expressing a complicated, multistep By using bullet points, it’s much easier
Use bullet process in paragraph form makes it to see what the steps are and when they
points difficult to understand the relationships are applied.
between steps.
Using Multiple Methods | Angela Colter

12. Expert review findings
At times, gave suggestions for improvements
Guidelines: If at any time during any rolling consecutive twelve billing
cycle period you fail to make two Minimum Payments on a
Eliminate timely basis or exceed your Credit Limit twice we may elect
jargon to increase your Purchase, Cash Advance and/or Balance Original (101 words)
Transfer APRs to the Penalty APRs. All Penalty APRs will
Use short remain in effect until, in a subsequent rolling consecutive
words six billing cycle period, you do not exceed your Credit Limit
at any time and you make all of your required Minimum
Use short Payments on a timely basis when, in your next billing cycle,
all Penalty APRs will no longer apply.
sentences
If you pay late or go over your credit limit twice in a
year, the interest rate you pay on most things goes up to Rewrite (48 words)
the default rate. It will go back down when you pay on
time and don’t go over your credit limit for six months.
Using Multiple Methods | Angela Colter

13. Usability test
Tested the material with 12 users
Gave them two sets of tasks using two different
documents: one solicitation letter, one card member
agreement
Topics covered:
• Annual percentage rates • Cash advance fee/APR
• Grace period • Balance transfer fee/APR
• Minimum payment • Finance charges/balance
• Late fee computation method
• Over limit fee • Changes to terms
• Payment allocation
Using Multiple Methods | Angela Colter

14. Usability test plan
Can they use disclosures Question Correct answer
to locate information?
Do they interpret the info
correctly once found?
Using Multiple Methods | Angela Colter

15. Usability test findings
Gave an overview citing Task 8: Finance Charges / Balance Computation
Method
task success rate Solicitation Letters: Only 1 of 12 located the information and
seemed to know the significance of the term average daily
Followed with a full balance. No one knew what two-cycle ADB meant.
explanation of outcome, Cardmember Agreements: All were able to locate the
information, but only one was able to correctly interpret the
information she read.
observations and
This was, by far, the task with the lowest success rate.
participant quotes Only one out of the 12 participants located the name of the
balance computation method used by the credit card in the
solicitation letter. Two participants said things like “I know
it’s based on the average daily balance, but that's not what
it’s called.” Part of the problem may have been the label
used on the Schumer box: Balance Calculation Method for
Purchases didn’t seem to convey “this is how we calculate
the interest you owe.”
Only one person gave a plausible explanation for the
definition of what Average Daily Balance Computation
Method means. No one who was asked what Two-Cycle
Average Daily Balance Computation Method means even
tried to guess.
Using Multiple Methods | Angela Colter

16. Usability test findings
Detailed information and performance metrics were
included in the report’s appendix.
Question P# Doc Time Correct? P’s Answer Analysis
This card uses P6 SL #1 0:15 N -- You pay interest on this
the “two-cycle month’s and last
average daily P8 SL #1 0:50 N “I can’t month’s balance.
balance” understand it”
It means if you made
method to
a partial payment last
figure out how P7 SL #2 1:35 N “I really don’t
month, you are paying
much interest know”
interest on principle
you owe. What
you have already paid
does it mean? P9 SL #2 -- N “You got me”
off.
P1 SL #3 1:10 N Described No one got it right.
averaging
P3 SL #4 0:16 N “No clue”
Using Multiple Methods | Angela Colter

17. Complimentary approach
Findings Weakness
Readability 12th grade reading level Doesn’t tell you what to do
formulas (but P.L. guidelines do)
Plain Docs don’t provide hierarchy, Doesn’t mean users will
language group related information, have problems
review use markers (TOC, (but usability testing shows
meaningful headings), or use where they do)
simple words and sentences
Usability test Users had trouble locating Small sample size can be a
info and often didn’t hard sell
understand it once they had (but video puts a face on
located it. problems; findings from
formulas and P.L. review
Disincentives for reading
found some of the same
terms.
issues identified in testing)
Using Multiple Methods | Angela Colter

18. Outcomes
Communicating concepts wasn’t the main problem,
unfair practices were.
The U.S. Congress and Federal Reserve passed laws
and regulations that:
• ban universal default and “two-cycle” billing
• require high interest balances be paid first
• require opt-in to over limit fees
• disclose consequences of making only the minimum
payment on monthly bills, not just in disclosures.
Using Multiple Methods | Angela Colter

19. Further reading
GAO Report: CREDIT CARDS Increased Complexity in
Rates and Fees Heightens Need for More Effective
Disclosures to Consumers
http://www.gao.gov/new.items/d06929.pdf
A Plain English Handbook: How to create clear SEC
disclosure documents
http://www.sec.gov/news/extra/handbook.htm
Using Multiple Methods | Angela Colter

20. Questions?
Angela Colter
Email: acolter@gmail.com
Twitter: @angelacolter