OPTIMIZATION QA:
How to Avoid
Launching Tests that
Break Your Site
When you break shit you
pollute data, lose money
and ev...
@joeljharvey #convcon
Goal
@joeljharvey #convcon
What’s QA?
@joeljharvey #convcon
Let’s say that …
@joeljharvey #convcon
30% of your
visitors
get a broken
experience on
your site
@joeljharvey #convcon
Y
O
U
R
SITE IS
BROKEN ON ONE OF
T
H
E
S
E
SCREENS
@joeljharvey #convcon
No One On Your
Team Knows
@joeljharvey #convcon
20% RPV
IE BUG
@joeljharvey #convcon
INTERNET
EXPLORER
@joeljharvey #convcon
You’re about to hire this guy as your CPA
@joeljharvey #convcon
QA MATH
@joeljharvey #convcon
Internet Explorer Traffic: Feb 2013
Internet Explorer Traffic: Feb 2014
@joeljharvey #convcon
 120% Increase Internet Explorer (IE)
Revenue
 20% Increase in All Revenue
 20% Increase in IE Av...
@joeljharvey #convcon
Who Are
You?
@joeljharvey #convcon
Who Am I?
- COO & Managing Partner,
Conversion Sciences
- Conversion Scientist
- Data Geek
- Entrepr...
@joeljharvey #convcon
RE-DEFINING QA
#1
@joeljharvey #convcon
STOP STEALING FROM YOURSELF
@joeljharvey #convcon
30% of your
visitors
get a broken
experience on
your site
QA DRIVES
REVENUE
@joeljharvey #convcon
OLD BORING
DEFINITION
@joeljharvey #convcon
@joeljharvey #convcon
QA is just a Cost
Center
We can’t
support
everything!
QA prevents us
from being agile
People on really old
devices probabl...
QA IS A
MARKETING
FUNCTION
@joeljharvey #convcon
QA STRATEGY
#2
@joeljharvey #convcon
INFUSE QA INTO YOUR OPTIMIZATION DNA
@joeljharvey #convcon
1. CULTURE
3. TECH PARADIGMS
2. REAL USER USE CASES
QA IS
OPTIMIZATION
@joeljharvey #convcon
1. Understand your visitors
@joeljharvey #convcon
2. Make sure control is not
broken
@joeljharvey #convcon
3. Generate great hypotheses &
test them
@joeljharvey #convcon
@joeljharvey #convcon
4. Don’t
pollute
your test
data
5. Do what the data tells you to
do
@joeljharvey #convcon
BUILDING A
QA CULTURE
@joeljharvey #convcon
@joeljharvey #convcon
#QAHAMMER
@joeljharvey #convcon
Build Destroy
HAMMER UP
@joeljharvey #convcon
BUY IN
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
20% RPV
IE BUG
@joeljharvey #convcon
REAL USE
CASES
@joeljharvey #convcon
90%
@joeljharvey #convcon
VISITORS
DON’T USE
YOUR SITE THE
WAY YOU
THINK THEY DO
@joeljharvey #convcon
@joeljharvey #convcon
SCREEN RECORDING
@joeljharvey #convcon
USER TESTING
DOCUMENT
AND QA ALL
USE CASES
@joeljharvey #convcon
YOUR JOB IS TO
BREAK YOUR
SITE
@joeljharvey #convcon
@joeljharvey #convcon
WHAT
SHOULD YOU
SUPPORT?
@joeljharvey #convcon
5%
@joeljharvey #convcon
@joeljharvey #convcon
Old New
BROWSER SUPPORT
 Internet Explorer 8
 Chrome 7
 Firefox 4
 Safari 4.1
 iOS 5.1 *
 Inte...
@joeljharvey #convcon
OS SUPPORT
Old New
 Windows XP
 OS X 10.4 (Tiger)
 Android 2.3
 iOS 5.1
 Windows Phone 7
 Wind...
5% MINIMUM
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
QA TOOLS
#4
@joeljharvey #convcon
BUILD YOUR QA’TION STATION
@joeljharvey #convcon
SEE YOUR
SITES LIKE
YOUR
VISITORS
iOS 7.1?
@joeljharvey #convcon
@joeljharvey #convcon
SOFTWARE
vs.
HARDWARE
@joeljharvey
#convcon
iPad,
iOS 5
iPad,
iOS
7.1
Windows
Vista
Android 4.3
iPhones:
iOS4, 5, 7
Android:
2.3.6, 4.0.4,
Windo...
@joeljharvey #convcon
@joeljharvey #convcon
?
@joeljharvey
#convcon
1. Browser
2. Operating System
3. “Other”
- Visitor Internet Speed
- Browser Plugins
- Flash
- Resol...
QA PROCESS
#4
@joeljharvey #convcon
BE THE HAMMER
QA
EXPERIENCES,
NOT PAGES
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
@joeljharvey #convcon
TEST QA
PROCESS
@joeljharvey #convcon
1.
Build Your Test
@joeljharvey #convcon
2.
Use Cookie, Query
String and/or Location
Targeting so Only You
See Experience
@joeljharvey #convcon
3.
Pause / Disable All
But One Experience
@joeljharvey #convcon
4.
Review Live
Experience In All
Supported
Browser/OS Combos
(like REAL USERS)
@joeljharvey #convcon
5.
Clear Cookies, Pause
Good-To-Go
Experience and
Unpause the Next One
@joeljharvey #convcon
6.
Repeat
@joeljharvey #convcon
A FEW TIPS
RANDOMIZE
YOUR
APPROACH
CAUTION WITH
CLASSES
WATCH SCRIPT
STACKS
FEAR THE
WYSIWYG
OWNERSHIP
@joeljharvey #convcon
Need to Borrow a QA Hammer?
Call 1-888-961-6604
and mention this
session for a chance
to win a Free ...
Upcoming SlideShare
Loading in …5
×

QA: How I Learned to Stop Breaking and Start Loving Tests

928 views

Published on

Joel Harvey details new ways to look at quality assurance and outlines the approach his company, Conversion Sciences ensures it's tests don't break client websites

Published in: Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
928
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
1
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

QA: How I Learned to Stop Breaking and Start Loving Tests

  1. 1. OPTIMIZATION QA: How to Avoid Launching Tests that Break Your Site When you break shit you pollute data, lose money and eventually get fired. Here’s how to avoid that pain and look like a hero in the process. @joeljharvey #convcon Joel Harvey, COO Conversion Sciences
  2. 2. @joeljharvey #convcon Goal
  3. 3. @joeljharvey #convcon What’s QA?
  4. 4. @joeljharvey #convcon Let’s say that …
  5. 5. @joeljharvey #convcon 30% of your visitors get a broken experience on your site
  6. 6. @joeljharvey #convcon Y O U R SITE IS BROKEN ON ONE OF T H E S E SCREENS
  7. 7. @joeljharvey #convcon No One On Your Team Knows
  8. 8. @joeljharvey #convcon 20% RPV IE BUG
  9. 9. @joeljharvey #convcon INTERNET EXPLORER
  10. 10. @joeljharvey #convcon You’re about to hire this guy as your CPA
  11. 11. @joeljharvey #convcon QA MATH
  12. 12. @joeljharvey #convcon Internet Explorer Traffic: Feb 2013 Internet Explorer Traffic: Feb 2014
  13. 13. @joeljharvey #convcon  120% Increase Internet Explorer (IE) Revenue  20% Increase in All Revenue  20% Increase in IE Average Order Value  167% Increase in IE Conversion Rate $1,500,000 Per Year
  14. 14. @joeljharvey #convcon Who Are You?
  15. 15. @joeljharvey #convcon Who Am I? - COO & Managing Partner, Conversion Sciences - Conversion Scientist - Data Geek - Entrepreneurial MBA
  16. 16. @joeljharvey #convcon
  17. 17. RE-DEFINING QA #1 @joeljharvey #convcon STOP STEALING FROM YOURSELF
  18. 18. @joeljharvey #convcon 30% of your visitors get a broken experience on your site
  19. 19. QA DRIVES REVENUE @joeljharvey #convcon
  20. 20. OLD BORING DEFINITION @joeljharvey #convcon
  21. 21. @joeljharvey #convcon
  22. 22. QA is just a Cost Center We can’t support everything! QA prevents us from being agile People on really old devices probably aren’t buyers QA is an IT / Development Task @joeljharvey #convcon
  23. 23. QA IS A MARKETING FUNCTION @joeljharvey #convcon
  24. 24. QA STRATEGY #2 @joeljharvey #convcon INFUSE QA INTO YOUR OPTIMIZATION DNA
  25. 25. @joeljharvey #convcon 1. CULTURE 3. TECH PARADIGMS 2. REAL USER USE CASES
  26. 26. QA IS OPTIMIZATION @joeljharvey #convcon
  27. 27. 1. Understand your visitors @joeljharvey #convcon
  28. 28. 2. Make sure control is not broken @joeljharvey #convcon
  29. 29. 3. Generate great hypotheses & test them @joeljharvey #convcon
  30. 30. @joeljharvey #convcon 4. Don’t pollute your test data
  31. 31. 5. Do what the data tells you to do @joeljharvey #convcon
  32. 32. BUILDING A QA CULTURE @joeljharvey #convcon
  33. 33. @joeljharvey #convcon #QAHAMMER
  34. 34. @joeljharvey #convcon Build Destroy
  35. 35. HAMMER UP @joeljharvey #convcon
  36. 36. BUY IN @joeljharvey #convcon
  37. 37. @joeljharvey #convcon
  38. 38. @joeljharvey #convcon 20% RPV IE BUG
  39. 39. @joeljharvey #convcon
  40. 40. REAL USE CASES @joeljharvey #convcon
  41. 41. 90% @joeljharvey #convcon
  42. 42. VISITORS DON’T USE YOUR SITE THE WAY YOU THINK THEY DO @joeljharvey #convcon
  43. 43. @joeljharvey #convcon SCREEN RECORDING
  44. 44. @joeljharvey #convcon USER TESTING
  45. 45. DOCUMENT AND QA ALL USE CASES @joeljharvey #convcon
  46. 46. YOUR JOB IS TO BREAK YOUR SITE @joeljharvey #convcon
  47. 47. @joeljharvey #convcon
  48. 48. WHAT SHOULD YOU SUPPORT? @joeljharvey #convcon
  49. 49. 5% @joeljharvey #convcon
  50. 50. @joeljharvey #convcon Old New BROWSER SUPPORT  Internet Explorer 8  Chrome 7  Firefox 4  Safari 4.1  iOS 5.1 *  Internet Explorer 11  Chrome 33  Firefox 27  Safari 7.0  iOS 7.1 *
  51. 51. @joeljharvey #convcon OS SUPPORT Old New  Windows XP  OS X 10.4 (Tiger)  Android 2.3  iOS 5.1  Windows Phone 7  Windows 8  OS X 10.9 (Maverick)  Android 4.4  iOS 7.1  Windows Phone 8.1
  52. 52. 5% MINIMUM @joeljharvey #convcon
  53. 53. @joeljharvey #convcon
  54. 54. @joeljharvey #convcon
  55. 55. @joeljharvey #convcon
  56. 56. @joeljharvey #convcon
  57. 57. @joeljharvey #convcon
  58. 58. @joeljharvey #convcon
  59. 59. @joeljharvey #convcon
  60. 60. @joeljharvey #convcon
  61. 61. @joeljharvey #convcon
  62. 62. QA TOOLS #4 @joeljharvey #convcon BUILD YOUR QA’TION STATION
  63. 63. @joeljharvey #convcon SEE YOUR SITES LIKE YOUR VISITORS
  64. 64. iOS 7.1? @joeljharvey #convcon
  65. 65. @joeljharvey #convcon SOFTWARE vs. HARDWARE
  66. 66. @joeljharvey #convcon iPad, iOS 5 iPad, iOS 7.1 Windows Vista Android 4.3 iPhones: iOS4, 5, 7 Android: 2.3.6, 4.0.4, Windows 8, no updates OS X 10.6 Windows 8, regular updates + VM’s OS X 10.8 OS X 10.9 OS X 10.7 + VM’s Windows XP Windows 7
  67. 67. @joeljharvey #convcon
  68. 68. @joeljharvey #convcon ?
  69. 69. @joeljharvey #convcon 1. Browser 2. Operating System 3. “Other” - Visitor Internet Speed - Browser Plugins - Flash - Resolution QA’TION STATION CHECKLIST
  70. 70. QA PROCESS #4 @joeljharvey #convcon BE THE HAMMER
  71. 71. QA EXPERIENCES, NOT PAGES @joeljharvey #convcon
  72. 72. @joeljharvey #convcon
  73. 73. @joeljharvey #convcon
  74. 74. @joeljharvey #convcon
  75. 75. TEST QA PROCESS @joeljharvey #convcon
  76. 76. 1. Build Your Test @joeljharvey #convcon
  77. 77. 2. Use Cookie, Query String and/or Location Targeting so Only You See Experience @joeljharvey #convcon
  78. 78. 3. Pause / Disable All But One Experience @joeljharvey #convcon
  79. 79. 4. Review Live Experience In All Supported Browser/OS Combos (like REAL USERS) @joeljharvey #convcon
  80. 80. 5. Clear Cookies, Pause Good-To-Go Experience and Unpause the Next One @joeljharvey #convcon
  81. 81. 6. Repeat @joeljharvey #convcon
  82. 82. A FEW TIPS
  83. 83. RANDOMIZE YOUR APPROACH
  84. 84. CAUTION WITH CLASSES
  85. 85. WATCH SCRIPT STACKS
  86. 86. FEAR THE WYSIWYG
  87. 87. OWNERSHIP
  88. 88. @joeljharvey #convcon Need to Borrow a QA Hammer? Call 1-888-961-6604 and mention this session for a chance to win a Free QA Audit ($15,000 Value) Conversion Conference San Francisco, 2014 Joel Harvey, Conversion Sciences Joel@ConversionSciences.com

×