0
A Crash Course in A/B Testing on Mobile
Websites
#optimizemobile | 20 November, 2014

Friday, November 22, 13
A Crash Course in A/B Testing on Mobile
Websites
#optimizemobile

#optimizemobile
Friday, November 22, 13
EXCLUSIVE WEBINAR GIVEAWAY
Win a free signed copy of A/B Testing!

How to Enter
Tweet what you’ve
learned today, and tag i...
Agenda
1. How the right mobile strategy can
accelerate your marketing ROI
2. A/B test ideas for mobile web design
principl...
How the right mobile
strategy can
accelerate your
marketing ROI

Friday, November 22, 13
#optimizemobile
Friday, November 22, 13
What does it mean to be mobile?

•mobile website?
•mobile app?
•both?

#optimizemobile
Friday, November 22, 13
Mobile websites
matter. What’s the next step?

#optimizemobile
Friday, November 22, 13
Deliver the
mobile web experience
that your users deserve.
Build

#optimizemobile
Friday, November 22, 13

or

Buy
Separate mobile (and tablet) sites
Your visitor is redirected to the right
version of your website, based on what
device t...
Responsive web design
Your website
automatically adapts
its content and layout
to fit your visitor’s
screen size or device...
What content should
your mobile website
have?
We have some (test) ideas...

#optimizemobile
Friday, November 22, 13
Mobile users
want to complete
tasks and find
information.

lhw.com

#optimizemobile
Friday, November 22, 13

Shop

Search
...
Device-Specific Presentation
(not device-specific content)

youtube.com

#optimizemobile
Friday, November 22, 13

m.youtube....
How do you define your
mobile website goals?

#optimizemobile
Friday, November 22, 13
Checkout
Flows
Test | Exit routes,
process flow,
security
messaging
Goal | More
purchases or
greater average
order values
...
Store Locators

Main Menu

Top Navigation

lululemon.com

perfumania.com

Goal | Drive in-store purchases
#optimizemobile
...
Layouts for
Ad and
Content
Engagement

Fixed Banner Ad
m.huffpost.com

#optimizemobile
Friday, November 22, 13

Banner Ad
...
Cross-Channel
Attribution

#optimizemobile
Friday, November 22, 13

Image source: Google
A/B test ideas
for mobile web
design
principles

Friday, November 22, 13
Is your mobile
website fast to
load and
fast to respond to
gestures?

#optimizemobile
Friday, November 22, 13
Hitting the Mobile Navigation Sweet
Spot

#optimizemobile
Friday, November 22, 13
Below the Fold
m.rei.com

Above the Fold
m.rei.com

#optimizemobile
Friday, November 22, 13

Test | Breadcrumb
and CTA
pla...
What gets more
page views?
• Buttons
• Back button
What gets more
clicks?
• Sticky button
• Above the fold
ideeli.com

gar...
Users scroll more on touch
screens.

#optimizemobile
Friday, November 22, 13

Image source: ClickTale
What compels
you to scroll?
• Lists
• Grids
• Text-heavy
blocks
• Images

huffingtonpost.com

nytimes.com

#optimizemobile...
#optimizemobile
Friday, November 22, 13
Test | Search functionality
Goal | Inform search
prominence

bonobos.com

#optimizemobile
Friday, November 22, 13
What leads to
more searches?
• Auto-complete
• Form fields
• Position on the
page

christies.com

#optimizemobile
Friday, ...
Simplification of Elements

youtube.com

#optimizemobile
Friday, November 22, 13

m.youtube.com
Test | Simplify
category page
elements like
filtering and
sorting
Goal | Identify
best presentation
on initial page
load
m...
What makes it
easier to find
what you’re
looking for?
• Default order
• Sorting options
• Filtering
options
stelladot.com
...
Menu Location and Patterns

Fly-out menu
garmin.com

#optimizemobile
Friday, November 22, 13
Test | Menu location and
patterns
Goal | Evaluate how
integrated menu experience
should feel (full screen vs.
layered on c...
Menu Location and Patterns

Blinds

amazon.com

#optimizemobile
Friday, November 22, 13

Buttons

m.timhortons.com

Blocks...
How to use
A/B testing
to drive
conversions
through the
roof
Friday, November 22, 13
Visual Editor

#optimizemobile
Friday, November 22, 13

Toggle between
<divs>
HTML + CSS + JS
edits

Code Editor

Manual i...
• Clicks + Page Views + Custom Events (Specified
with JS)
• Goals may/not directly impact end conversion
#optimizemobile
F...
• Positive/negative targeting conditions
• Cookie, referral source, campaign id, custom tags,
location

#optimizemobile
Fr...
Best Testing Practices
1.
2.
3.
4.

Friday, November 22, 13

Test with a hypothesis.
Sequence your testing.
Test broad UX ...
Results from
a real mobile
A/B test

Friday, November 22, 13
Case Study |
Test | Add paid
featured listings to
mobile search
results
Goal | Validate
value of desktop
functionality on
...
Case Study |

Result | +1.6% lift in
new
leads generated

Variation B Wins!
#optimizemobile
Friday, November 22, 13
EXCLUSIVE WEBINAR GIVEAWAY
Win a free signed copy of A/B Testing!

How to Enter
Tweet what you’ve
learned today, and tag i...
GET A FREE MOBILE
ASSESSMENT
visit mobify.com/go/mobile-assessment
for full details
#optimizemobile
Friday, November 22, 1...
Any Questions
for Ben and Khattaab?

A Crash Course in A/B Testing on Mobile
Websites
#optimizemobile
Friday, November 22,...
Upcoming SlideShare
Loading in...5
×

A Crash Course in A/B Testing on Mobile Websites

1,424

Published on

Most marketers we’ve spoken to say “never,” the reason being that they just don’t know where to start.

If you feel this way too, join Mobify and Optimizely for a “crash-course” webinar to help you finally get started with A/B testing on your mobile website for increased engagement and maximized conversions.

In one hour, you’ll learn:

-How the right mobile strategy can accelerate your marketing ROI.
-Key mobile web design elements that improve conversions.
-How to use mobile A/B testing to drive conversions through the roof.

Published in: Technology, Design
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,424
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
57
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • Ben Terrill, VP Customer Success at Mobify.
    With a decade-long background in digital agencies, Ben has created award-winning campaigns for brands like Nike and Electronic Arts. His experience both as an executive leader and front-line web developer drives Mobify’s professional services team to continually deliver excellence to all of Mobify’s customers, including British Telecom, Starbucks and Expedia.
    Khattab Khan, Strategic Optimization Consultant at Optimizely.
    Khattaab’s experience with UX design and A/B testing across industries equips him with best practices to guide customers through test ideation and iteration, sequencing, results review, and program management.
  • Ben Terrill, VP Customer Success at Mobify.
    With a decade-long background in digital agencies, Ben has created award-winning campaigns for brands like Nike and Electronic Arts. His experience both as an executive leader and front-line web developer drives Mobify’s professional services team to continually deliver excellence to all of Mobify’s customers, including British Telecom, Starbucks and Expedia.
    Khattab Khan, Strategic Optimization Consultant at Optimizely.
    Khattaab’s experience with UX design and A/B testing across industries equips him with best practices to guide customers through test ideation and iteration, sequencing, results review, and program management.
  • ubiquity of mobilestatsMobile traffic is growing at a rate of 1.5x per year and as of May 2013 mobile traffic represents 15% of Internet traffic (Meeker report, slide #32). In developing markets this trend is even higher, for instance in China % of people accessing internet from mobile phones surpassed those from desktop PCs (75% vs 71%) (Meeker report, slide 33)Globally there are over 1.5B smartphone users with a YoY growth rate of 31%.  219M in the US with a penetration rate of 58%. (Meeker, #40) 58% of people who own smartphones have used them for store-related shopping. In 2012, almost 25% of all online Black Friday shopping came from phones and tablets. (from quicksprout inforgraphics) (feel free to use more as it makes sense)
    http://www.mobify.com/go/50-mobile-commerce-stats/
    check this post for some other numbers: http://blog.optimizely.com/2013/11/14/mobile-ab-testing-announcement/
  • - how do i get one?- what content should it have?- what are my mobile goals?- how do i optimize my mobile website so i’m achieving these goals?
    ----
    I want to take a moment to recognize the three different scenarios most companies find themselves in with regards to having a mobile-optimized website today. And mention, no matter where you’re at, there are opportunities for you with building an amazing web experience and with mobile A/B testing.
    Scenario 1: No mobile-optimized experience yet: Your visitor gets your desktop website no matter what device they’re using.
    - limited resources, budget. you know that you need a solution in order to stay competitive.
  • - how do i get one?- what content should it have?- what are my mobile goals?- how do i optimize my mobile website so i’m achieving these goals?
    ----
    I want to take a moment to recognize the three different scenarios most companies find themselves in with regards to having a mobile-optimized website today. And mention, no matter where you’re at, there are opportunities for you with building an amazing web experience and with mobile A/B testing.
    Scenario 1: No mobile-optimized experience yet: Your visitor gets your desktop website no matter what device they’re using.
    - limited resources, budget. you know that you need a solution in order to stay competitive.
  • there are many great vendors out there that will..
    - provide tools to help you build a website (Mobify included...)
  • Let’s look at three different scenarios most companies find themselves in with regards to having a mobile-optimized website today. And mention, no matter where you’re at, there are opportunities for you with building an amazing web experience and with mobile A/B testing.
    Scenario 1: Separate Mobile Site
    Your visitor is redirected to the right version of your website, based on what device they’re using.
    - if you’re in this boat, you likely created a separate website or worked with a vendor to create a mobile site.
    - Each version has a separate URL (so an m-dot for mobile, and t-dot for tablet), and a separate code base.
    - this was the best way to build mobile sites back in the day, but it has proven to present some issues when sharing links over social media and email, with SEO and when trying to get feature parity between channels.
  • Scenario 2. Responsive Web Design:
    Your website automatically adapts its content and layout to fit different screen sizes so your visitor always sees the best version of your website.
    - This is now the industry best practice
    - Building a responsive website from scratch takes a lot of resources, skill and time, but once it’s set up, the development process is streamlined (you only have to maintain a single website and code base for all devices), and the user experience is consistent across all devices.
  • Once you’ve got the framework in place, now you need the content. This is where testing comes in. How do you know how much content? how little? should it be the same as your desktop? If it is similar, how can you fit all that into such a small screen? Experimentation will help.
  • Leading Hotels of the World
    Misconception: mobile sites should deliver only task-based functionality, not information-seeking content
    Answer: mobile users require both; information-seeking (discovery) is a task
    ---
    Re: context
    Time of day (during commute 3g vs at home on wifi. distinctly different patterns based on context)
    We need to understand is when our customer wants to do what, and through which channel - and be available to them on their terms.
    - Desktop vs mobile: scrolling, carousels
    - smartphone vs tablet: browsing, purchasing (conversion rates lower on smartphone than on tablet)
    - ios and android (device types): distinct behavior patterns
  • What works on your desktop site, might not work on your mobile site.
    So we need to take mobile design best practices with your mobile website goals and experiment.
    Content and form are not intertwined
    Don’t create platform-specific content
    Content doesn’t have to look exactly the same between desktop and mobile
    Aim to direct users to most-used content at the same time you target desktop site functionality
    TEST
    Get user attention with conclusion first + easy to read bullets
    Headings should be links---
    - Direct to most used content- Aim for desktop functionality parity
  • The increased diversity of devices and accessibility to the web are making the customer’s journey more complex: 90% of people move between devices to accomplish a goal, whether that’s to make a purchase, download an app, or subscribe to a service. (source: Google)
    A big challenge for marketers right now is cross-channel attribution. We know that before making an online purchase decision, our customer may engage with our brand through many different media channels over several days. In this light, each and every touch point matters. So how do we best serve our customers at each interaction, to create an overall amazing experience?
    ---
    http://www.google.com/think/tools/customer-journey-to-online-purchase.html
  • * How do i optimize my mobile website so i’m achieving these goals?
    ---
    Before Khattab dives into how to use A/B testing to drive conversions through the roof, I’m going to walk through a few foundational elements of a high-converting mobile website.
  • We want a fast responsive website.
    fast to load, fast to respond to gestures.
    - carousels at the same velocity as your finger swipes.
    - experience feels native.
    - small nuanced touches make the experience feel faster and increase user engagement.
  • The ways users physically interact with mobile devices informs where on the screen to test
    Input type and grip should drive the placement of navigation controls, not screen size.
    Mobile: only about 1/3 of screen is within effortless thumb reach
  • CTA + breadcrumb placement, zoom activation, collapsed/exposed detail
    Goal | Drive engagement with content that compels CTA conversion
  • *khattab’s slide
  • Users are not completely averse to scrolling, especially in discovery mode
    Touch screens offer an even more natural scrolling experience, compelling further scroll
  • *khattab’s slide
  • *khattab’s slide
  • *khattab’s slide
  • *khattab’s slide
  • There are some elements a user should gather instantly from your web presence, such as branding recognize-ability, your company’s name, what the site is about.
  • [KK]  
    Menu options (from Optimizely Candyland blog post)
  • [KK]
    Screenshot of REI (left) with bullets of optimization opportunities (right)
  • [KK]  
    Menu options (from Optimizely Candyland blog post)
  • There are some elements a user should gather instantly from your web presence, such as branding recognize-ability, your company’s name, what the site is about.
  • There are some elements a user should gather instantly from your web presence, such as branding recognize-ability, your company’s name, what the site is about.
  • In this section,
  • Case study is rooted in migrating desktop functionality to mobile
    Big business gains from small lifts
    1.6% increase in conversions was extremely significant
    This metric validated the ROI of ApartmentGuide.com’s new search functionality and gave RentPath real data to back up business decisions around mobile
    Non-traditional conversion goals: optimization data can provide big picture insights on how your business is performing
    Results analysis: segmentation
    RentPath plans to use visitor segments to build targeted tests for each device
  • Case study is rooted in migrating desktop functionality to mobile
    Big business gains from small lifts
    1.6% increase in conversions was extremely significant
    This metric validated the ROI of ApartmentGuide.com’s new search functionality and gave RentPath real data to back up business decisions around mobile
    Non-traditional conversion goals: optimization data can provide big picture insights on how your business is performing
    Results analysis: segmentation
    RentPath plans to use visitor segments to build targeted tests for each device
  • Ben Terrill, VP Customer Success at Mobify.
    With a decade-long background in digital agencies, Ben has created award-winning campaigns for brands like Nike and Electronic Arts. His experience both as an executive leader and front-line web developer drives Mobify’s professional services team to continually deliver excellence to all of Mobify’s customers, including British Telecom, Starbucks and Expedia.
    Khattab Khan, Strategic Optimization Consultant at Optimizely.
    Khattaab’s experience with UX design and A/B testing across industries equips him with best practices to guide customers through test ideation and iteration, sequencing, results review, and program management.
  • Transcript of "A Crash Course in A/B Testing on Mobile Websites"

    1. 1. A Crash Course in A/B Testing on Mobile Websites #optimizemobile | 20 November, 2014 Friday, November 22, 13
    2. 2. A Crash Course in A/B Testing on Mobile Websites #optimizemobile #optimizemobile Friday, November 22, 13
    3. 3. EXCLUSIVE WEBINAR GIVEAWAY Win a free signed copy of A/B Testing! How to Enter Tweet what you’ve learned today, and tag it with #optimizemobile #optimizemobile Friday, November 22, 13
    4. 4. Agenda 1. How the right mobile strategy can accelerate your marketing ROI 2. A/B test ideas for mobile web design principles 3. How to use mobile A/B testing to drive conversions through the roof 4. Results from real mobile A/B tests 5. Q&A with Ben and Khattaab Friday, November 22, 13
    5. 5. How the right mobile strategy can accelerate your marketing ROI Friday, November 22, 13
    6. 6. #optimizemobile Friday, November 22, 13
    7. 7. What does it mean to be mobile? •mobile website? •mobile app? •both? #optimizemobile Friday, November 22, 13
    8. 8. Mobile websites matter. What’s the next step? #optimizemobile Friday, November 22, 13
    9. 9. Deliver the mobile web experience that your users deserve. Build #optimizemobile Friday, November 22, 13 or Buy
    10. 10. Separate mobile (and tablet) sites Your visitor is redirected to the right version of your website, based on what device they’re using. m.website.com t.website.com www.website.com #optimizemobile Friday, November 22, 13
    11. 11. Responsive web design Your website automatically adapts its content and layout to fit your visitor’s screen size or device. Always at www.website.com #optimizemobile Friday, November 22, 13
    12. 12. What content should your mobile website have? We have some (test) ideas... #optimizemobile Friday, November 22, 13
    13. 13. Mobile users want to complete tasks and find information. lhw.com #optimizemobile Friday, November 22, 13 Shop Search About Support Contact Q&A Links Gallery
    14. 14. Device-Specific Presentation (not device-specific content) youtube.com #optimizemobile Friday, November 22, 13 m.youtube.com
    15. 15. How do you define your mobile website goals? #optimizemobile Friday, November 22, 13
    16. 16. Checkout Flows Test | Exit routes, process flow, security messaging Goal | More purchases or greater average order values #optimizemobile Friday, November 22, 13 “Begin Checkout” bose.com “Secure Checkout” ctshirts.com
    17. 17. Store Locators Main Menu Top Navigation lululemon.com perfumania.com Goal | Drive in-store purchases #optimizemobile Friday, November 22, 13
    18. 18. Layouts for Ad and Content Engagement Fixed Banner Ad m.huffpost.com #optimizemobile Friday, November 22, 13 Banner Ad wired.com
    19. 19. Cross-Channel Attribution #optimizemobile Friday, November 22, 13 Image source: Google
    20. 20. A/B test ideas for mobile web design principles Friday, November 22, 13
    21. 21. Is your mobile website fast to load and fast to respond to gestures? #optimizemobile Friday, November 22, 13
    22. 22. Hitting the Mobile Navigation Sweet Spot #optimizemobile Friday, November 22, 13
    23. 23. Below the Fold m.rei.com Above the Fold m.rei.com #optimizemobile Friday, November 22, 13 Test | Breadcrumb and CTA placement, zoom activation, collapsed or exposed detail Goal | Drive engagement with content that compels CTA conversion
    24. 24. What gets more page views? • Buttons • Back button What gets more clicks? • Sticky button • Above the fold ideeli.com garmin.com #optimizemobile Friday, November 22, 13
    25. 25. Users scroll more on touch screens. #optimizemobile Friday, November 22, 13 Image source: ClickTale
    26. 26. What compels you to scroll? • Lists • Grids • Text-heavy blocks • Images huffingtonpost.com nytimes.com #optimizemobile Friday, November 22, 13
    27. 27. #optimizemobile Friday, November 22, 13
    28. 28. Test | Search functionality Goal | Inform search prominence bonobos.com #optimizemobile Friday, November 22, 13
    29. 29. What leads to more searches? • Auto-complete • Form fields • Position on the page christies.com #optimizemobile Friday, November 22, 13 lhw.com
    30. 30. Simplification of Elements youtube.com #optimizemobile Friday, November 22, 13 m.youtube.com
    31. 31. Test | Simplify category page elements like filtering and sorting Goal | Identify best presentation on initial page load m.rei.com m.stubhub.com #optimizemobile Friday, November 22, 13
    32. 32. What makes it easier to find what you’re looking for? • Default order • Sorting options • Filtering options stelladot.com ideeli.com #optimizemobile Friday, November 22, 13
    33. 33. Menu Location and Patterns Fly-out menu garmin.com #optimizemobile Friday, November 22, 13
    34. 34. Test | Menu location and patterns Goal | Evaluate how integrated menu experience should feel (full screen vs. layered on content) m.rei.com #optimizemobile Friday, November 22, 13
    35. 35. Menu Location and Patterns Blinds amazon.com #optimizemobile Friday, November 22, 13 Buttons m.timhortons.com Blocks ideeli.com
    36. 36. How to use A/B testing to drive conversions through the roof Friday, November 22, 13
    37. 37. Visual Editor #optimizemobile Friday, November 22, 13 Toggle between <divs> HTML + CSS + JS edits Code Editor Manual inputs
    38. 38. • Clicks + Page Views + Custom Events (Specified with JS) • Goals may/not directly impact end conversion #optimizemobile Friday, November 22, 13
    39. 39. • Positive/negative targeting conditions • Cookie, referral source, campaign id, custom tags, location #optimizemobile Friday, November 22, 13
    40. 40. Best Testing Practices 1. 2. 3. 4. Friday, November 22, 13 Test with a hypothesis. Sequence your testing. Test broad UX themes. Set goals with indirect impact.
    41. 41. Results from a real mobile A/B test Friday, November 22, 13
    42. 42. Case Study | Test | Add paid featured listings to mobile search results Goal | Validate value of desktop functionality on mobile Variation A Variation B #optimizemobile Friday, November 22, 13
    43. 43. Case Study | Result | +1.6% lift in new leads generated Variation B Wins! #optimizemobile Friday, November 22, 13
    44. 44. EXCLUSIVE WEBINAR GIVEAWAY Win a free signed copy of A/B Testing! How to Enter Tweet what you’ve learned today, and tag it with #optimizemobile #optimizemobile Friday, November 22, 13
    45. 45. GET A FREE MOBILE ASSESSMENT visit mobify.com/go/mobile-assessment for full details #optimizemobile Friday, November 22, 13
    46. 46. Any Questions for Ben and Khattaab? A Crash Course in A/B Testing on Mobile Websites #optimizemobile Friday, November 22, 13
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×