The document summarizes a study evaluating the mobile banking experiences of major New Zealand banks and one large US bank. It assessed the banks based on 5 criteria: platforms supported, features, user ratings, usability, and interaction design. Based on weighted scores across all criteria, ASB and the large US bank JP Morgan Chase had the top mobile banking experiences, while ANZ had the lowest score due to few features, platforms, and poor user ratings. The study found that good mobile usability and features can drive customer switching between banks.
4. Mobile and mobile
banking were hot trends in
NZ in 2012 (penetration:)
http://services.google.com/fh/files/blogs/our_mob
ile_planet_new_zealand_en.pdf
TNS Mobile Life study (New Zealand) 2012
5. Google search trends told the same story:
Mobile
banking
(NZ)
Banking
(NZ)
Google Analytics
6. By mid-2012 it was like a mad race with New Zealand banks
announcing new mobile banking offerings on a weekly basis!
8. Global mobile stats in 2012…
6 billion mobile connections
1.2 billion mobile web users
1 billion smartphone handsets
http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats/a#subscribers
http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats/a#smartphonepenetration
9. Rank Country Subscribers
(millions)
1 China 1,100
2 India 699
3 USA 322
4 ? 260
Where is mobile 5 Brazil 259
growing? 6 Russia 227
7 Japan 128
8 ? 120
9 Germany 114
http://mobithinking.com/mobile-
marketing-tools/latest-mobile-
stats/a#chinasubs
10 ? 107
10. Rank Country Subscribers
(millions)
1 China 1,100
2 India 699
3 USA 322
4 Indonesia 260
Where is mobile 5 Brazil 259
growing? 6 Russia 227
7 Japan 128
8 Pakistan 120
9 Germany 114
http://mobithinking.com/mobile-
marketing-tools/latest-mobile-
stats/a#chinasubs
10 Nigeria 107
11. What about growth in +2000%
mobile banking?
1.1bn
550m
55m
2009 2013 2015
http://www.berginsi http://www.ababj.com/tech- http://www.prweb.co
ght.com/News.aspx? topics-plus/m-banking-530- m/releases/2010/02/pr
m_m=6&s_m=1 million-users-by-2013-2695.html web3553494.htm
12. China 2012:
>1 billion mobile subs
http://mobithinking.
>400m mobile web m/blog/china-top-
mobile-market
users
13. The global trend is for “un-banked” and “under-banked”
consumer segments driving the growth in mobile banking
http://mobithinking.com/banking-the-unbanked
14. Q: This country has 29
million mobile
subscribers.
65% (19 million) use
mobile money services.
Where am I?
A:
http://www.cck.go.ke/resc/statcs.html
15. Q: This country has 29
million mobile
subscribers.
65% (19 million) use
mobile money services.
Where am I?
A: Kenya
http://www.cck.go.ke/resc/statcs.html
16. And it turns out it’s the same trend in
the US as in China, India and Kenya….
17. The “unbanked” and
“underbanked” are key to
growth – adoption among
younger, minorities, lower
income segments are driving
growth in the US
18. But the question we really wanted to
answer…who provides the BEST mobile banking
experience in New Zealand?
19. So, UX consultancies regularly study the usability of
internet and mobile banking…
22. So we asked ourselves
- as UX researchers
and designers - if we
were to evaluate
mobile banking
offerings as objectively
and scientifically as
possible, …how would
we do it?
23. This is how. Evaluate each offering against 5 core criteria:
24. We assessed all New Zealand banks (and US giant) and
plotted them on a pentagon spider-diagram
Usability
Interaction
Platforms
design
User
Features
ratings
25. Big graph area = good
Usability
Interaction
Platforms
design
User
Features
ratings
26. Small graph area = bad
Usability
Interaction
Platforms
design
User
Features
ratings
27. Before we start, however, it’s important to understand
the relative size and resources of the banks we
evaluated…
TSB Bank ~$200m
revenue
JP Morgan Chase
$100bn+ revenue
28. We started with some secondary research – looking at
publically available data
Platforms
User ratings Features
29. It turned into this heat-map…
iPhone
Platform
Y Y Y Y Y Y y y
iPad N Y N N N N y y
Android Y Y Y Y Y N y y
Windows Y N N N N N y N
Open accounts N N Y N N Y N N
No log-in balances Y Y Y N N N N N
4/5 digit log-in Y N Y Y Y N N Y
View account balances Y Y Y Y Y Y Y Y
View transaction lists Y Y Y Y Y Y Y Y
Transfer between accounts Y Y Y Y Y Y Y Y
Bill payment Y Y Y N Y Y Y N
Set up payees - From Mobile N N N N N Y Y N
View automatic payments N Y Y N Y Y N Y
Change upcoming payments N N Y N Y y N N
View term deposits Y Y Y N Y y N N
View mortgages & loans Y Y N Y Y Y Y Y
Functionality
View KiwiSaver Y N N N N N N N
Pay tax N N N N Y Y N N
Alerts: Text N N N N N Y Y N
Push N N N N N Y Y Y
Email N N N N N Y Y N
Foreign exchange, rates etc. Y N N N Y Y N N
Share trading N N N N N N N y
NFC N N N N N y N N
Location based services Y N Y N N N Y Y
Pay to mobile Y N N Y N Y N Y
Pay TradeMe sellers Y N Y N N N N N
Pay to Facebook friends Y N N N N N N N
Personalise account N N N N N Y N N
Personalised service N N N N Y N N N
Picture cheque deposits N N N N N N Y N
Loyality scheme points Y n N N N N Y N
Secure Messaging Y N N N Y Y N N
App store - latest version 4.53 3.33 4.55 1.82 4.35 4.45 2.34 2.19
Rating
App store - all versions 3.31 1.87 4.16 1.82 4.35 4.45 3.49 3.00
Google play 3.94 2.93 4.50 3.93 4.35 4.24 3.92
30. We looked closely at the features provided – both core
and non-core
Features
31.
32. Here is how they ranked on features…tiny TSB at the top!
33. Next we looked at the user rating in the app stores
(which must be taken with a grain of salt…)
User
ratings
48. In the end, ANZ achieved the lowest score due to a lack of
features, few platforms and poor user ratings
Usability
56.01
Interaction
Platforms
design
User ratings Features
49. ASB, by contrast achieved strong scores across all criteria
Usability
77.83
Interaction
Platforms
design
User ratings Features
50. In fact, giant US Bank JP Morgan Chase (often held up as a
benchmark in mobile banking) came in second to ASB
Usability
74.86
Interaction
Platforms
design
User ratings Features
52. Finally, some key insights from this study…
• Is “feature bloat” a problem in terms of mobile
banking UX? Not if the app is well designed.
• It’s really hard to measure good design!
• Good mobile usability and features can make
people switch banks!
• Americans still use cheques!
• And, these findings are already out of date…
Editor's Notes
This is some of us…We have 20 guys who do user experience strategy, research and design in Auckland and Wellington. Some of us.We’ve completed more than 1,000 UX projects for 200 clients spanning 20 industries since 2002 [CLICK]
Banking is our single biggest client segment – so this is both credentials and disclaimer!In fact, we’ve completed more than 80 UX projects between Westpac and ASB (CBA’s Kiwi subsidiary) alone – they are both Top 5 clients for usBut over time we’ve worked with all the Big 4 NZ banks and some of the smaller ones as well
Mobile banking was the hot trend in NZ in 2012. Here are some numbers:According to Google, smartphone penetration in NZ reached 44%And a global TNS Mobile Life study claimed that 28% of Kiwi’s already use mobile banking
This is evident in Google search trends as well
In fact, mid-way through the year it was like a mad race with banks announcing new mobile banking apps and features on a weekly basis!And as user experience consultancy with a big banking practice we needed by make sure we stayed ahead of the pack and had a clear grasp of the latest trends and intel from the world of mobile bankingSo we invested a lot of time looking over the horizon – and we found out some fascinating stuff!
Lets start with the really big pictureThere are 6 billion mobile subscriptions globally. Growth is in India and China – already 30% of global subscribers. Mobile subscribers outnumber fixed line connections 5:11.2 mobile user access the internet on their phone1 billion people globally have a smartphone
Lets start with the really big pictureThere are 6 billion mobile subscriptions globally. Growth is in India and China – already 30% of global subscribers. Mobile subscribers outnumber fixed line connections 5:11.2 mobile user access the internet on their phone1 billion people globally have a smartphone
Ok, time for a quick pop quiz: this is the Top 10 list of countries by number of mobile subscribers in 2012.Any guesses as to who is 4th on the list?
Nigeria
So lets get back to the global picture for mobile banking:Bergh Insights estimated 55 million users in 2009Juniper Research forecasts 550 users by 2013Global Industry Analysts predict 1.1bn users by 2015That’s a healthy 2000% growth over 6 years. So who’s driving this growth? [CLICK]
Its there guys:In 2012, the number of mobile subscribers in China passed 1 billion…and 400m Chinese accessed the internet on their mobile [click]
And its these guys. This is a picture of an actual “bank branch” in India. The global trend is for “un-banked” and “under-banked” consumer segments driving the growth in mobile banking.Makes a lot of sense, since here are more than twice as many people globally with mobile phones than with bank accounts in the world [CLICK]
Time for another Pop Quiz:Question [CLICK] AnswerSo what about developed economies? What is a solid source of truth about mobile banking here? [CLICK]
Time for another Pop Quiz:Question [CLICK] AnswerSo what about developed economies? What is a solid source of truth about mobile banking here? [CLICK]
How about these guys – the US Federal Reserve! They research and monitor the mobile banking space closely. This is a research report from March 2012.One of their key findings was around who is driving the growth in mobile banking in the US [CLICK]
Turns out its these guys:And this effectively the same trend as in China, India and Kenya. It’s “unbanked” and “underbanked” driving the growth – adoption among younger, minorities, lower income segments are driving growth in the US. The Fed goes so far believes that mobile is “closing the digital divide” in the US
One option is to look at what other user experience companies think. We are member of the UX Alliance – who regularly evaluates internet and mobile banking offeringsBut these studies don’t really point to who provids the BEST mobile banking experience
There are a bunch of organisations and associations handing out mobile banking awards. Keynote is one of the more prominent ones. They announced that Chase Bank was the global leader in mobile banking in 2012But they didn’t really focus on the mobile banking experience as such…
We were quite excited to this – the 2012 Mobile Banking Awards! – but it seems this is just one guys opinion, based mostly on other people’s options!Worse still – Michael Nuciforo actually worked on the project to deliver the ANZ goMoney app that he has given second place in his own awards.This stuff goes against our religion. As a UX consultancy – everything we do is as scientific as we can make it – based on primary research. Wherever possible we make inferences based on observed behaviours and objective data
So we asked ourselves – if we were evaluating mobile banking offerings as objectively and scientifically as possible – bearing in mind that our domain is the user experience - how would we do it?KN: At this point I’m going to hand over to my own mad scientist - Michael Dutton. Thanks Michael…MD: Thanks Kris, let me talk you through how we went about doing this – and what we found [CLICK]
We settled on five evaluation aspects that combined gave us a robust, objective user experience rating:Firstly, what users can actually do on the mobile app [clicks]What platforms can the app be used on – iPhone, Android, iPad and Windows [clicks]We looked at the user ratings for all the apps in the Apple App Store and Google Play markets[clicks]We asked eight (?) UX consultants and designers to individually evaluate each banking app against a set of interaction design and usability criteria[clicks]But most importantly, we user tested all the apps in our testing labs in Auckland and Wellington[clicks]Maria, heath, sharon, KA, natalie, alex, mark, mdd
We assessed every New Zealand mobile banking app and plotted them on a five-sided spider diagram. Some of the New Zealand apps are basically poor cousins of the Australian “parent” versions.We also made the call to evaluate [one] mobile banking app that is recognised as one of the best in the world – [JP Morgan Chase] - to make sure we didn’t get to Kiwi-introspective.I am a customer of Chase bank, which made it easy to get access.
Better mobile banking offering = bigger diagram area
Worse mobile banking offering = smaller area
Lets quickly put into context the relative resources of the banks we’ve looked at…ANZ = $7bn revenue. Westpac, BNZ and ASB are roughly the same size at $4bn-ish. KiwiBank is about $1bn. And TSB is “only” $200 millionCompare this with Chase bank which turns over $100 billion +
We ended up with this sort-of “uber-spread-sheet-heat-map“ across these three criteriaLet me break this down a bit for easier consumption
In 2012, Verve Mobile conducted a survey asking users whatare the most valuable services for a financial app. 63% transfers62% bill pay (Two parts to this, setting up a payee and actually paying a bill)40% Alerts (Three parts to this; text, email, push options – can you do it via mobile?)33% Person to Person payments29% Branch ATM LocatorWe took those five features and added a sixth one; finding a past transaction. Sounds simple and it was, but our participants wanted to know more about the transaction and what their running balance was at that point in time. Why would they want this? More details = greater priming to aid in memory recall. Running balance = reduces cognitive burden.
As you can see, implementing those features comes in many different concepts, most interestingly is Kiwi Banks Pay and Transfer icon that looks like Pie for some reason.Most banks had the pay/transfer features explicit and upfront like ASB, but when we look at Chase, a lot of participants didn’t know what the marketing speak “Chase Quickpay” meant. Why isn’t paying bills or transfering money, quick? Top right you’ll see Planned Payments. Essentailly it’s pending payments that you requested a future based date on. Of course one of the greatest assests to having a smartphone device is the GPS integration allowing you to find anything anywhere. And of coursse on the bottom right here we have the ability to deposit a check via photography. Tell a Kiwi about this neat feature and they look at you with astonishment; “you use checks in america?”We established in total 26 features that mobile banking could have. This is based on the core six identified by Verve, along with 20 features banks had already introduced in some form. So how many is too many?
We’ll answer that in a bit. As you can see, TSB has the greatest number of features, followed by ASB. At the bottom, we have WestPac and ANZ with only 3 core features each. Westpac has 5 non-core and ANZ has 3 non-core. Remember the 2006 HBR study on the number of product features and it’s impact on usability? They found that 66% of people will go for the feature rich item, but after using the item, only 44% would actually use it. It will be interesting to see if that held true to our research, six years later!
What do ratings tell us? For the app maker it’s a quick and easy way to view the direction of your latest release. For the consumer it’s really difficult to know what a rating means. It’s self reported information and sometimes not even real reporting information; the defunked YourAppReviewed dot com use to post hundreds of five star reviews for money. That being noted, real reviews do provide some interesting feedback. [click]
One bad app release can seriously impact an app’s rating. Greater market penetration, greater the impact. Look at Chase. All versions, 3.5 stars, current version had two bugs according to user comments and was rated 2.3. Out of the 507 reviews of their current app, 269 rated it 1 star!!!!!There is no starndard for user ratings: “Support iPhone 5 please”, “iPhone 5 support ASAP, other than that is good”. Which one was rated three Stars and the other was rated only 1 Star!Ratings also vary by iTunes store, NZ versus USA.
There’s not much to say about platforms, we know the four major: iOS, android, windows and tablet (adroid/ios). We this round of research we simply stuck with iPad for tablet.
Random chat about this. Speaks for its self.
What:[CLICK] In a 2012 survey of users regarding most desired mobile banking features, Transfers (63), Transactions, Bill Pay (62), Alerts (40), Person to Person payments (33), and ATM/Branch locator (29), were the rated the most desired features. We took this as a sign of what to test! Locked apps to whatever was published in the app stores on January 15th, 2013. Who: [CLICK]Millenials, why? According to Deliotte those born from 1979 – 1995 are the young adult consumers that represent the fastest growing segment of today’s workforce and 25% of the global population. Most active mobile banking users. Expected annual income is to reach 3.4 trillion by 2018. Where? [CLICK] We did the testing in Auckland and Wellington. Method: [CLICK]Traditional lab testing, iPhone 4 device with all the apps installed, counter balanced design of course, single individual capturing all data for consistency. Participants could not test their own bank, captured SUS and Stack Ranking of favorite to least favorite app tested.
[What we found]
[What we found]
Poorly implemented features, definitely impact SUS scores and appear to correlate to average # of usability issues identified by participant. Define SUS. [CLICK]Remember the HBR study on feature and it’s impact on usability We found a similar effect here, TSB had the most features and the most issues, yet it ranked dead last as the app that people would choose out of those they tested! What does this tell us? There is no magic number on the limit of features, but the greater the number of features the more important good design is in making those features usable.
While you look at this graph, Let’s just be clear, this is just a data point, it doesn’t mean they won or have the best user experience!We didn’t want to just take a SUS score and show it to you, so we captured the average issue score which is the average number of usability issues (regardless of severity), and we also captured users stacked ranking of preference which when combined with the SUS score is determining our Uber Usability score. TSB: 19 featuresWestPac: 8 featuresASB: 16 featuresWestPac-3.33 ---- 85 ---- 5.3 = 90.30ASB-1.22 ---- 83.44 ---- 6.9 = 90.34ANZ-2.00 ---- 81.25 ---- 6 = 87.25BNZ-3.13 ---- 80 ---- 6 = 86KiwiBank-5.10 ---- 70.31 ---- 7.5 = 77.81Chase (BM)-6.22 ---- 58.53 ---- 4.2 = 62.73TSB-13.38 ---- 44.38 ---- 1 = 45.38
IxD experts. We all fall into the trap, it’s better because of X. Translate that from tacit knowledge to objective and evaluative process. Heuristics become dated, quickly. An underlined link on a mobile app, just isn’t relevant. Relevant Framework: What’s important in mobile banking: Security, aesthetic effects – we are dealing with multi-million dollar banks, affective computer matters, efficiency, bringing up relevant keyboards or custom keyboards for numeric entry points. Be objective, but don’t limit your IxD experts. Their design knowledge translates well into psychological principles that can be objective. Be patient, comparison and consistency takes diligence.Our secret sauce: Well we are only showing part of it, but in all its consists of:Areas of focus: 10Criteria: 50
Starting and Finishing: from logging in the first time to logging outNavigation and IA: labeling, structure, navigation modelErrors: if you encountered any, messaging, clarity, understanding.Accessibility: Let’s face it, for most of us, everyday our eyes are deterorating just a bit. Therefore these apps should adhere to established Accessibility principlesComprehension: as Kris pointed out mobile banking is taking off in under-developed countries. Simple and meaningful words, clarity of iconography, reduce comprehension and the gateway to usage.Efficiency of use: isn’t it the bankers who like to say, Time is money?Layout: Lack of clutter, clean, intuitive, efficientVisual Design: aesthetics are important as it establishes brand, but also aids in navigation, efficiency, IA, etc. General: technical issues, security (auto-matic logout), basic help and support, confirmation messaging as appropriate.
So we have these five areas that we created to measure the apps against, but we realized it’s not fair to weight them equally. Not deploying to a platform can be a business decision, especially if your customers aren’t there. The user ratings, they can be helpful for finding problems, but they can also be falsified in today’s world. Interaction design, that’s definitely important, but how do you assess that interaction design in the real world; through usability testing. So we weighted our Uber Usability score a bit higher and we also looked at the features. Just because you have 19 features doesn’t mean you are meeting the basic feature needs of your market. So we put a weighting on core features, the ones that the users asked for.
So, how did it all come out in the wash? [CLICK]
The worst performing of all the NZ banking apps was ANZ with an over-all score of 56.01 out of 100 [CLICK]While ANZ score well in both Usability and Interaction Design, the ANZ offering is well behind in basic features, platform coverage and app store user ratings. User RatingUber UsabilityCore FeatureIxDPlatformsANZ36.487.2533.3375.9240
The best performing of all the NZ banking apps was ASB with an over-all score of 77.83 out of 100 [CLICK]While ANZ score well in both Usability and Interaction Design, the ANZ offering is well behind in basic features, platform coverage and app store user ratings. User RatingUber UsabilityCore FeatureIxDPlatformsASB90.690.3455.5680.7880
For comparison, Chase bank came in 2:nd over-all
So here is the winners’ podium – the Top 3 performers in our study [CLICK]With an honorouble mention for Westpac as the 3rd best New Zealand mobile banking app
Feature bloat is acceptable as long as it doesn’t impair usabilityThat creating a robust and objective framework for IxD is challengingThat good usability along with offering core features is an incentive for individuals to switch banksKiwi’s hand out account numbers like business cards. American’s still hand out check’s . . . I don’t know why Chase bank, perhaps next year you’ll do better