2. Contents
• Executive summary
• Usability testing protocol
• Logistics
• Scenario overview
• Task success rates
• Detailed review of testing results
• Post-test questionnaire results
• Participant comments
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 2
3. Executive Summary
Overall
• The designs performed well
• The issues discovered were minor and will be addressed as design
progresses
• Advisor and staff participants discussed pain points for the current client
registration process. The most common challenge revolves around
remembering/knowing client ID. All felt that the newer simplified process will
help
The testing indicated good usability of the design
• All participants liked the clearer layout, better location/size of the components
and the bigger fonts
• All said that they would prefer to have a way to reset their password rather
then calling in for support
• Iconography did not test well with participants. Participants did not notice the
“eye” nor the “lock.” When asked, they did not know what the icons meant
or what it would do
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 3
4. Usability Testing Protocol
Usability is "The extent to which a product can be used by specified users to achieve
specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.“
- Usability Standards ISO 9241
A usability lab reviews the following to assess how well a piece of technology will
support or prevent a user from completing their task:
• Learnability: How easy is it for users to accomplish basic tasks the first time they
encounter the design?
• Efficiency: Once users have learned the design, how quickly can they perform tasks?
• Memorability: When users return to the design after a period of not using it, how easily
can they re-establish proficiency?
• Errors: How many errors do users make, how severe are these errors, and how easily
can they recover?
• Satisfaction: How pleasant is it to use the design?
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 4
5. Logistics
December 5th- 7th – Chicago
Evaluations conducted in a professional research facility
14 participants:
3 AFG advisors
2 AAG advisor
3 AFA
2 Staff
1 Client
3 Consumers
Devices tested
3 iPads
8 Desktop
3 Mobile phones
60 minute sessions
All sessions observed and recorded
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 5
6. Scenarios and Tasks – Client/Consumer
Scenario 1: Register to view your Ameriprise account information (Registration)
1. Find register button
2. Complete personal information and create a username/password. No security
questions are created.
3. Review completion page
Scenario 2: View your Ameriprise account information (Login from trusted site)
1. Login
2. Launch fake Secure Site accounts page
Scenario 3: Forgot password (Forgot/Reset password)
1. Login
2. Locate Forgot password link
3. Select verification type (text)
4. Reset password
5. Review completion page
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 6
7. Scenarios and Tasks – Client/Consumer
Scenario 4: Set security questions (MFA)
1. Login
2. Set security questions and answers
3. Review completion page
4. Feedback on the types of questions
Scenario 5: View your Ameriprise account information from tax preparer’s office
(Login from non-trusted site)
1. Login
2. Select verification type (security question)
3. Answer question
4. Review completion page
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 7
8. Scenarios and Tasks – Advisor/Staff
Scenario 1: View Advisor Compass home page (Login from trusted site)
1. Login
2. Launch fake Advisor Compass page
Scenario 2: Forgot password (Forgot/Reset password)
1. Login
2. Locate forgot password link
3. Request verification type (text)
4. Reset password
5. Review completion page
Scenario 3: Set security questions (MFA)
1. Login
2. Set security questions and answers
3. Review completion page
4. Feedback on the types of questions
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 8
9. Scenarios and Tasks – Advisor/Staff
Scenario 4: View your Ameriprise account information from home (Login from
non-trusted site)
1. Login
2. Select verification type (Security question)
3. Answer question
4. Review completion page
Scenario 5: Q & A about client registration process
1. Processes to get client registered
2. Difficulties
3. Feedback they’ve been hearing or have encountered themselves
4. Showed some participants the client registration process for feedback
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 9
10. Task Success Rates – Client/Consumer
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 10
Scenarios/Tasks Pass Fail Assisted Did not
complete
Average
completion
time
1) Register for Secure Site
- Find register button 75% 25%
10 min- Complete steps 1-3 100%
- Review confirmation page 100%
2) View Ameriprise accounts
- Login 100% 1.5 min
- View fake accounts page 100%
3) Forgot/Reset password
- Locate Forgot password link 100%
4.75 min
- Select verification type (text) 100%
- Reset password 100%
- Review confirmation screen 100%
4) Set Security questions
- Login 100%
5.5 min- Set security questions/answers 100%
- Review confirmation screen 100%
5) Login from non-trusted device
- Login 100%
5.5 min- Select/Answer verification type 100%
- Review confirmation page 100%
- Existing uplifted
functionality
performed well and
fast
11. Task Success Rates – Advisor/Staff
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 11
Scenarios/Tasks Pass Fail Assisted Did not
complete
Average
completion
time
1) View trusted site
- Login 100% 2.8 min
- View Advisor Compass 100%
2) Forgot/Reset password
- Locate Forgot password link 75% 25%
4.5 min- Select verification type (text) 100%
- Reset password 100%
- Review confirmation screen 100%
3) Set Security questions
- Login 100% 5.4 min
- Set security questions/answers 100%
- Review confirmation screen 100%
4) Login from non-trusted
device
- Login
100%
5.5 min
- Select/Answer verification type 100%
- Review confirmation page 100%
- Existing uplifted
functionality
performed well and
fast
- Finding the “forgot
user ID” link was
problematic for one
user
12. Registration process (Client)
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
12
- All were concerned about not
knowing or where to find their client
ID if it was not provided to them
Consider an alternate option of asking
SSN versus a client ID
13. Registration process (Advisor)
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
13
- All advisors/staff reacted positively to
removing the MFA questions at
registration.
- 2 participants worried what would
happen if the user encountered issues
with login prior to setting them
Advisor comments on barriers to client registration:
- Older clients don’t trust technology or know how to use
- Remembering passwords is an issue
- Many clients would rather print, sign and store documents for themselves
- Advisors can’t see what the client “sees” so they don’t know what the client is
looking at and can’t help them
- Client’s don’t understand what a client ID is
Suggestions made by advisors and staff:
- Make it simpler for the client. They don’t understand all the steps involved
- Create a “demo” for advisors to be able to walk the client through the process
- Document how secure it is and benefits of having online access in “client friendly”
terms
14. Login to trusted site
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
14
- All participant liked the size and use of white space
on the screen
- Only one participant noticed the “Remember User
ID”. When asked, all said that they liked this added
feature and would use it with their own devices.
15. Forgot/Reset password
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
15
- All participants typed in username first
before clicking the “Forgot password”
link
- All but one participant found the link
right away.
16. Forgot/Reset password
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
16
- All participants using a touch
device had issues when going
from an active field to an inactive
field (global issue)
17. Forgot/Reset password cont’d
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
17
- All participants liked that there were
multiple ways to retrieve the
password.
- 4 participants were unsure of where
the phone number came from and
were worried that it might not be
correct or a land-line.
- 2 participants were confused about
the two different numbers in the text
retrieval mock-up.
- 3 participants mentioned that they
would like to eventually see finger
scan capability
Consider:
1) exploring pre-populating the User ID
on next screen if the user typed it in
on initial Login screen
2) only have one number in the SMS
text retrieval alert
18. Set security questions
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
18
- Participants felt that the ”point in time” questions would
be harder to remember as tastes change (favorite
movie/actor/music group)
- All felt that having these security questions gave an
added security to the site
- All said that they would be able to choose and
remember at least one from each of the categories
- Only 2 wanted the option to create their own questions
- 3 commented that they never know if the answer is
case sensitive or not
Consider:
1) adding on screen text about case sensitivity
2) removing favorite or point in time and replace with
more factual questions
19. Set security questions cont’d
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
19
- Answering the security questions were problematic for
users testing on the iPhone.
- Location of the dropdown was not seen by 2 of
the 3 participants
- Due to the length of the questions, wording was
cutoff so they were not able to see the full
question
Consider:
1) not using the browser default for the dropdowns and
replace with another way to display the dropdowns
contextually near the field
2) shorten the questions to be able to see in view port (in
small form factor only)
20. Login from non-trusted site
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 20
- Only 2 participants did not
understand why they were being
asked to verify their identity
- All understood the trusted device
list checkbox
- All participants thought the “Try
another” link would ask them
another security question, not to
change the method of
verification.
Consider:
1) renaming the link label
2) adding the ability to change the
question as well as the changing
the method to another
verification method
21. Iconography
For internal use only. Not intended for inspection by, or distribution or quotation to the general public.
21
- Only one participant understood the
functionality of the “eye”. All said that
they would use it, but mainly when
they were having issues with the
password and wanted to confirm they
typed it in correctly
- Only 2 understood the lock icon
- When showing the “eye” functionality
with participants, users did not know
to click the icon twice. The first click
initiated the tool tip and users did not
know to click again to activate the
password
For user recognition, consider:
1) using text versus an ”eye” icon
2) removing the “lock” icon
22. Questionnaire Results
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 22
Questions Did not
answer
1= not completely
usable, useful or
likely
2 3=average 4 5=completely
usable, useful or
likely
Rate the “ease of use” of
the capabilities shown
20% 80%
Confidence in how
Ameriprise is adapting it’s
applications to work across
multiple devices
30% 30% 40%
- Participants put “ease of use” a
little above average or completely
useable, useful or likely
- Confidence in Ameriprise’s
adaptation of multiple devices is
split:
- 30% average
- 30% a little above average
- 40% likely
23. Questionnaire Results cont’d
For internal use only. Not intended for inspection by, or distribution or quotation to the general public. 23
Websites with good registration processes:
- Ones that the e-mail addresses as a user ID
- Chase Bank
- “My client’s often complain that they miss UBS online services. So easy to find statements, make a .pdf, very
easy to use and could get performance information online.”
- Vanguard
Which functionalities did they like best:
- Being able to see your password when typing
- Initial sign in box. Larger and more visually comfortable
- Reset password with text
- “The idea of a verification code is great. How about receiving it through phone call?”
- “Loved the 3-step security question”
- Multiple ways to retrieve password
- The show password
- Likes the green checkmarks so I’m certain I’m doing it right
Which functionalities did they like least:
- “Try another” wording because it was not clear meaning
- Tough to read security questions (on phone)
- Requiring client ID. Stating it’s on your statement isn’t helpful before getting a statement
- Eyeball icon was confusing
- “Some security questions were too vague”
- Would like to tab between user ID/password
24. Comments
“Don’t bother looking for something unique, it doesn’t need to be complicated.”
Steve S. – Consumer
“Remember user ID, I think is new. That’s nice. I normally do that on my
iPad.”
Cory G - AFA
“Huge win to be able to register on a phone. People are always using their
phone.”
Cory G - AFA
“I’m really pleased. Ameriprise has come a long way with technology.”
Cory G – AFA
“Ameriprise caters to younger and middle age clients. But you want to make it
easier for older people too.”
Lana R. – Ameriprise client
25. Comments
“Overall, the changes to the mobile app look great!”
Petra K. – Staff member
“Standard operating procedure, this isn’t a big deal. This is part of what goes
with logging on to any website.”
Lisa W. – Staff member
“Aesthetically pleasing and user friendly.”
Unknown – Post-test survey
“I actually liked your registration process because it felt easy, yet secure.”
Unknown – Post-test survey
