Responsive Design Tested: What a recent experiment reveals about the potential ROI of mobile design
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Responsive Design Tested: What a recent experiment reveals about the potential ROI of mobile design

on

  • 1,913 views

To most web designers and forward-looking experts in the field, responsive mobile design is not an optional feature. Mobile users have to be able to view your website on their devices without having ...

To most web designers and forward-looking experts in the field, responsive mobile design is not an optional feature. Mobile users have to be able to view your website on their devices without having to scroll everywhere to read it.

But is responsive design really worth the investment? Has anyone actually pitted a responsive design against a regular desktop design?

These were the questions in the minds of the marketers at a large media company faced with the mobile dilemma.

To answer these questions, the marketers designed a test.

Statistics

Views

Total Views
1,913
Views on SlideShare
1,853
Embed Views
60

Actions

Likes
6
Downloads
49
Comments
2

1 Embed 60

https://twitter.com 60

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Hey Mike,

    Thanks again for taking time to share your perspective with me. I wanted to reply-all here so everyone could get the benefit of some of the additional points of clarity we discussed and perhaps join in on the discussion.

    NOTE: Though I am not allowed to share certain pieces of data (per the partner), I will share below as much as I can per my conversations with the data sciences and page development teams.

    -The content on both control and treatment versions were exactly the same EXCEPT the control’s desktop background image (faux pop-up) and the control's desktop exit page link image (“x” versus button). The reason why these desktop variables were not completely eliminated was to minimize unnecessary development time and cost (running an 'in-between test') that could be completely mitigated by advanced test tracking and segmentation (which was used)

    -The test was actually tracked as 3 separate tests (mobile vs mobile, tablet vs tablet, desktop vs desktop) to increase the transferability of the learnings and to isolate the background image variable only present in the desktop control experience

    -The results of all 3 separate tests were aggregated (as shown in the initial slides) to provide an additional layer of executive level insight on the potential risk of a wide implementation of the responsive code (which was in favor of responsive design since mobile and tablet didn’t hurt them) as opposed to producing and implementing dedicated device pages (which would increase cost significantly when executed to multiple areas of the site)

    -The only change in code for the treatment were the CSS changes necessary to enable the responsive display of content according to screen size calling on the page.

    -Page weight was closely compared and page load time was tested on multiple devices and connections by a dedicated QA team and no significant difference was discovered or noted (thus limiting the impact of that additional, important variable). I would need to conduct a detailed interview with the QA team to be overly certain.

    -Sample distribution for each device category was consistent with historical audience data and behavior gleaned from a detailed data/metrics analysis and sample size was sufficient in each segment to provide a result that could be taken seriously from a scientific perspective (can’t share absolute numbers though..).

    -Since the content of the page (other than the background and link image) did not change, the way in which the form elements were set up and the keyboard types that were called on for each device category did not change, so it wasn't a variable in this test. I would need to do further investigation to determine which options were used (to see if a follow-up test would make sense on this variable)

    -Dedicated tablet and form pages (separate pages and code sets) were not tested in this protocol, specifically to understand the implications of using responsive (one code set) as opposed to multiple code sets and pages (as this has an impact on tech spend in scale). There may be a follow-up in the queue, but it is not high priority given the potential cost/benefit implications compared to other tests lined up that would produce a higher yield (potential gains vs scaled implementation cost)

    -On user research: Real-time audience behavior, trends from a data/metrics analysis over a long period of time were used to plan and anticipate the traffic for the test. Data from the test indicated a consistency of audience, mitigating a potential selection effect (i.e. the audience profile that makes up the site was consistent with the audience that participated in the test).

    -On additional UX style user studies and focus groups: I would have to do additional research to see if this was a factor. I do know that when it has been available to partnerships I have overseen, it has always been utilized to form the most informed hypothesis possible

    Hopefully this helps. All great observations and questions from Mike!

    Sincerely,

    Jon Powell
    Sr. Manager, Research and Strategy
    MECLABS
    Are you sure you want to
    Your message goes here
    Processing…
  • I was on the webinar and listened to the whole thing and unfortunately almost every aspect of the this presentation, it’s assumptions, the test itself and the interpretation of the findings are flawed. About the only thing correct was the description that responsive changes the layout of displayed content based on screen size.

    The minute the responsive design didn’t exactly match the non-responsive design for desktop you killed the confidence of the findings. Beyond that, the test suggests that responsive itself plays a role in conversion and it really doesn’t. Well designed websites with great content and user-friendly interactions do, regardless of responsiveness.

    During the talk the speaker mentioned the change in the design was because the existing design couldn’t be exactly replicated in responsive which is simply not the case. When the decision was made to change the design a new baseline for a non-responsive version of the new design need to be established to make the responsive findings of any value.

    There are also other factors not discussed that would dramatically impact the results.

    1. Were both versions, responsive and non-responsive, of equal page weight? In other words, was the code downloaded by the devices the same size in Kb for both designs? Download speeds have big impact on conversions and a poorly executed responsive site can easily weigh in at more than a non-responsive page.

    2. Were the numbers based on a similar sample size? Were there basically the same number of phone, tablet, and desktop visitors in both tests? Can you share the analytics?

    3. What was the overall sample size used for this experiment? Small sample numbers can yield seemingly huge shifts but actually be statistically irrelevant. So many missing data points that could have given confidence tot the findings.

    4. Were best practices followed when setting up the form elements. Did the responsive design make sure to assign the proper type to each form field to enable the most appropriate keyboard for different information, like phone numbers or emails? Data entry is more difficult and time consuming on touch devices and enabling the right keyboard facilitates completion. To be fair, I can’t really see what the forms are asking for so this might not be relevant here.

    5. You don’t mention testing a dedicated phone or tablet size design. It could simply be that this particular audience just doesn’t like to fill in forms on touch devices. Or, they may have had an entirely different goal while browsing from their phone? Was any direct user research done to find out if this audience prefers to use their desktop for form entry?

    In general users can’t tell a responsive site from a non-responsive site per se. What they can tell is if a site is slow loading, difficult to navigate, has poor content, looks untrustworthy, looks designed to fit their screen and many other things, but not if they are looking at a responsive site.

    A better test is one to test a poorly done responsive site vs. a well done responsive site.

    I’d caution everyone that heard this talk or is looking at the results shown and take them with a grain of salt. Sad thing is that I’ve been looking for some serious data on the matter, either for or against responsive, but all I keep finding stuff tike this.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Early Bird pricing for Email Summit is ending tomorrow,, but we wanted to give you a chance to take advantage of even more savings before this pricing special ends. We want to give web clinics attendees an extra $200 off your email summit ticket purchase -This includes $200 off the early bird price, giving you a total savings of $500! Don’t miss out on this exclusive offer. Visit MarketingExperiments.com/200 to learn more.

Responsive Design Tested: What a recent experiment reveals about the potential ROI of mobile design Presentation Transcript

  • 1. Responsive Design Tested What a recent experiment reveals about the potential ROI of mobile design
  • 2. We’re sharing on Twitter! #WebClinic
  • 3. Today’s speakers Austin McCraw Jon Powell Senior Editorial Analyst MECLABS Senior Manager, Research and Strategy MECLABS
  • 4. What is responsive design? Unresponsive Responsive
  • 5. What is responsive design? The MECLABS Conversion Index C = 4m + 3v + 2(i - f) - 2a© Friction: Psychological resistance to a given element in the conversion process, typically in the form of: • Unnecessary length • Unnecessary difficulty
  • 6. ? What impact does responsive design have on friction experienced by the customer?
  • 7. Experiment: Background Experiment ID: TP1933 Record Location: MECLABS Research Library Research Partner: [Protected] Background: A large news media organization trying to determine whether it should invest in responsive mobile design. Goal: To significantly increase the number of free trial sign-ups. Research Question: Which design will generate the highest rate of free trial sign-ups: responsive or unresponsive? Test Design: A/B multifactor split test
  • 8. Experiment: Control (unresponsive) Desktop *Android devices also included in this test Tablet
  • 9. Experiment: Control (unresponsive) Desktop *Android devices also included in this test Mobile
  • 10. Experiment: Treatment (responsive) Desktop *Android devices also included in this test Tablet
  • 11. Experiment: Treatment (responsive) Desktop *Android devices also included in this test Mobile
  • 12. Experiment: Side-by-side Unresponsive *Android devices also included in this test Responsive
  • 13. Experiment: Results 56% Aggregate increase in conversion The responsive design treatment showed an aggregate statistically significant difference in free trial subscriptions when compared to the control. Design Conversion Rate Control – Unresponsive 3.71% Treatment – Responsive 5.80% % Relative Change: *Android devices are also part of these results 56.41%
  • 14. ? Why did the responsive design outperform the unresponsive by so much?
  • 15. Experiment: Results 56% Aggregate increase in conversion The responsive design treatment showed an aggregate statistically significant difference in free trial subscriptions when compared to the control. Design Conversion Rate Control – Unresponsive 3.71% Treatment – Responsive 5.80% % Relative Change: *Android devices are also part of these results 56.41%
  • 16. Experiment: Results Design Conversion Rate 56% Aggregate Increase in Conversion Control – Unresponsive 3.71% The responsive design treatment showed an aggregate statistically significant difference in + Desktop trial subscriptions when compared to the control % free + Tablet % + Mobile Treatment – Responsive % 5.80% + Desktop % + Tablet % + Mobile % ? Which device makes the greatest contribution to the total difference?
  • 17. ! Today, we are going to review test data for each device category to determine its total impact on perceived friction.
  • 18. Desktop Performance Responsive design tested
  • 19. Experiment: Desktop control Unresponsive • For most screen sizes, all primary content is accessible without action • The primary content is framed within a faux pop-up style box over a faded background
  • 20. Experiment: Desktop treatment Responsive • Like the control, all primary content is accessible without action • The faux pop-up frame, however, is completely eliminated
  • 21. Experiment: Desktop side-by-side Unresponsive Responsive
  • 22. Experiment: Desktop results 105% Increase in conversion The responsive design desktop treatment showed a statistically significant difference in free trial subscriptions when compared to the control. Design Conversion Rate Desktop Control – Unresponsive 2.65% Desktop Treatment – Responsive 5.43% % Relative Change: 104.57%
  • 23. ? What was actually responsible for the difference in performance?
  • 24. Tablet Performance Responsive design tested
  • 25. Experiment: Tablet control Unresponsive iOS: • The majority of the content is accessible without action (supporting image requires scrolling) • A portion of the bottom of the page shows the pop-up Android: • All of the content is accessible without action; may require zoom for certain devices • Supporting imagery and aesthetic quality is lost in the page load
  • 26. Experiment: Tablet treatment Responsive iOS: • The supporting image is immediately visible and the popup background is completely eliminated Android: • Significant improvement in aesthetic and page load, though the supporting image is still lost • Only vertical scrolling required for ancillary content
  • 27. Experiment: Tablet side-by-side Unresponsive Responsive
  • 28. Experiment: Tablet results X No significant difference in conversion The responsive design tablet treatment showed no statistically significant difference in free trial subscriptions when compared to the control. Design Conversion Rate Tablet Control – Unresponsive 7.24% Tablet Treatment – Responsive 8.81% % Relative Change: 21.6%
  • 29. ? Why did the same treatments perform differently on different devices?
  • 30. Mobile Performance Responsive design tested
  • 31. Experiment: Mobile control Unresponsive iOS: • Requires scrolling to read complete sentences • Also requires scrolling to see all required fields and the primary call-toaction Android: • Requires one to two pinches to zoom movements and read the content • When zoomed in, requires horizontal scrolling to read complete sentences
  • 32. Experiment: Mobile treatment Responsive • Requires vertical scrolling to get to all fields as well as the call-to-action • Complete sentences are available without action • Generally a consistent experience across major device platforms
  • 33. Experiment: Mobile side-by-side Unresponsive Responsive
  • 34. Experiment: Mobile results X No significant difference in conversion The responsive design mobile treatment showed no statistically significant difference in free trial subscriptions when compared to the control. Design Conversion Rate Mobile Control – Unresponsive 5.14% Mobile Treatment – Responsive 4.34% % Relative Change: -15.6%
  • 35. Experiment: All results Design Conversion Rate % Rel. Change 3.71% - + Desktop 2.65% - + Tablet 7.24% - + Mobile 5.14% - 5.80% +56.4% + Desktop 5.43% +104.6% + Tablet 8.81% +21.6% Control – Unresponsive Treatment – Responsive + Mobile ? % Relative Change: 4.34% 56.41% -15.6% What is the customer wisdom? Why did the treatment win in aggregate?
  • 36. What you need to understand The MECLABS Conversion Index C = 4m + 3v + 2(i - f) - 2a © Friction The mobile and tablet treatments made significant changes in layout; they did not, however, make significant changes in friction experienced by the customer
  • 37. What you need to understand The MECLABS Conversion Index C = 4m + 3v + 2(i - f) - 2a © Friction Anxiety The desktop treatment did something slightly different The move to responsive design coincidentally reduced both friction and anxiety by eliminating the pop-up appearance of the control
  • 38. Responsive design tested F Key Principles 1. Responsive design, like any new technology or technique, does not necessarily increase conversion. This is because we are not optimizing design, we are optimizing thought sequences. 2. Responsive design is only as effective as the degree of change it affects in the mind of the customer. 3. This requires you to consider not only how responsive design will impact the look of the page, but the customers perception on two levels: • • Friction: How much will responsive design decrease perceived effort? Anxiety: How much will responsive design decrease perceived concern?
  • 39. EXCLUSIVE SAVINGS Save an EXTRA $200 on Email Summit Tickets Use promo code: 515-ST-1010 *Offer expires January 24
  • 40. Experiment: Background Experiment ID: TP1933 Record Location: MECLABS Research Library Research Partner: [Protected] Background: A large technology company selling software to small businesses. Goal: To significantly increase the number of software purchases. Research Question: Which design will generate the highest rate of software purchases? Test Design: A/B multifactor split test
  • 41. Experiment: Side-by-side Control COLUMN #1 Treatment COLUMN #2 COLUMN #1
  • 42. Which treatment won?
  • 43. To see the results … Live: Jan. 22 at 4:00 p.m. EST Watch live or get the full 35-minute video replay • What is the optimal number of columns for a page? • How should columns be weighted on a page? • Which is better: a right sidebar or left sidebar? To watch live or view the full video replay when it is available, register at the link below: MarketingExperiments.com/Columns
  • 44. See how you can conduct research with us MECLABS conducts rigorous experiments in the new science of optimization. We apply our discoveries to help leaders optimize the financial performance of their sales and marketing programs. Learn more about how you may be a fit for a MECLABS Research Partnership: • Select Research Partnership Opportunities on the postwebinar survey • Contact us directly info@MECLABS.com 1-877-635-0565 x