The document discusses how to properly conduct split-testing on a WordPress website. It defines split and multivariate testing as methods for testing different variations of web pages and elements. The author recommends establishing testing goals, sample sizes, and key performance indicators before testing. Various WordPress plugins are available to facilitate split testing. Common pitfalls to avoid include stopping tests prematurely and failing to account for changes in traffic or market conditions over time.

1. How to properly
split-test your
WordPress website
IGOR KOLOSOV
WORDCAMP VIENNA, APRIL 2015

2. What is split (A/B) testing?
o Traffic is split randomly and evenly between existing web page
(control version) and different variations of the same page
(Widerfunnel.com definition)
oA vs. B vs. n-th variation
oAdditional note: multiarmed bandit as a test type
@IGORKO_HUB ON #WCVIE, APRIL 2015

3. What is multivariate testing?
o Simultaneously testing numerous variations of the page by
combining web page elements (title, images, colours..)
o A1 B1 vs. A2 B1 vs. A1 B2 vs. A2 B2 etc.
@IGORKO_HUB ON #WCVIE, APRIL 2015

4. Why split/multivariate test?
o Data beats opinions
o Test what people really do and not what they say they would do
(in usability tests)
o Insights gained can help your other (marketing) efforts
o Valid Conversion rate optimization doesn’t exist without valid
testing
@IGORKO_HUB ON #WCVIE, APRIL 2015

5. Optimise or die
o Stepwise redesign (one big change every couple of years)
o Gains missed between launches
vs.
o Continuous design (and usefulness) improvements
o Requires more resources
o Website requirements changing fast so #2 is a better option in the longer run
o Establish process and the ROI won’t be a problem
Thanks Craig Sullivan (@optimiseordie) for the inspiration for the title 
@IGORKO_HUB ON #WCVIE, APRIL 2015

6. Before testing
o Quantitative analysis (answers what and how)
o Web analytics
o Behavior tracking and analysis (heatmaps, session recording)
o Qualitative analysis (answers why)
o User feedback
o Polls, interviews
o Define sample size and test length
o Use calculators for two-sided t-test
o The only way for the test to be statistically valid is to stick to this
@IGORKO_HUB ON #WCVIE, APRIL 2015

7. Before testing (continued)
o Seasonality or important dates
o Christmas sales, Cyber Monday, etc.
o Align with business strategy/goals
o determine KPIs
o make a testing roadmap
o Business cycles/waves length
o How many visits and days before the sale is made?
o Test for minimum one purchase cycle and/or two business cycles
@IGORKO_HUB ON #WCVIE, APRIL 2015

8. Before testing (continued)
o Technical check
o WP plugins compatibility (also consider removing unnecessary ones)
o Server resources
o Browsers, devices, screen resolutions
o Website performance (page load time)
@IGORKO_HUB ON #WCVIE, APRIL 2015

9. Before testing (resource list)
@IGORKO_HUB ON #WCVIE, APRIL 2015

10. Before testing (resource list
continued)
@IGORKO_HUB ON #WCVIE, APRIL 2015

11. What to test?
o Button? Background colour? Photo size or content of the photo? Headline?
Call-to-action on the button?
o Align with your goals first
o Don’t take other websites’ “good case practices“ for granted, but look for
patterns on highly optimized websites (Airbnb, Amazon, Booking.com…)
@IGORKO_HUB ON #WCVIE, APRIL 2015

12. How to test on WordPress?
o Page A vs page B (different URLs)
o Dynamic changes on page, which creates version A, B, C…
o Multivariate test – dynamic changes of elements within page
o Split test page template
o Split test website template
@IGORKO_HUB ON #WCVIE, APRIL 2015

13. Wordpress testing (resource list)
@IGORKO_HUB ON #WCVIE, APRIL 2015

14. General testing pitfalls and myths
o Stopping test as 95% significance is reached
o Sample size, business cycles, traffic volume is what determines test length
o Test results for one experiment are valid forever
o Market changes so as your visitor habits, traffic sources, promotion calls-to-action; re-do
one single test from time to time (rule of thumb: once a year)
o Test without conversion rate lift is failure
o Track and measure all the data and pull out conclusions: you will know what not to do any
more in the same way
@IGORKO_HUB ON #WCVIE, APRIL 2015

15. WordPress website testing
problems
o Often testing cannot run properly because of caching plugin setup
o Solution: try different caching plugin or turn of caching completely (use CDN
instead); sometimes excluding certain scripts from minification is enough
o Too many (unnecessary or poorly optimized?) plugins causing webpage to
load slow; combined with testing services becomes a problem
o Solution: turn off plugins that you don’t need
o Testing plugin not playing nice with some other plugin
o Solution: try alternative plugins if possible; try other testing service
@IGORKO_HUB ON #WCVIE, APRIL 2015

16. Other (resource list)
@IGORKO_HUB ON #WCVIE, APRIL 2015

17. About me & contact info
Web Analytics and Online business growth independent consultant, a psychologist working in the
field of internet marketing. Passionate about analysing website visitors’ behaviour, eliminating
frustrations and improving user’s online experience. Believes that randomness is just a name for an
undiscovered pattern.
Certified for Google Analytics and Adwords Search, Google Partner as an agency.
With 7+ years and a broad experience in CRO and web analytics, worked with more than 70 clients as
a freelance consultant. Former Customer Acquisition and Web analytics guy at ManageWP.
o contactme@igorkolosov.com
o igorkolosov.com
o http://optimize.rs
o Twitter: @igorko_hub
o LinkedIn: http://rs.linkedin.com/in/igorkolosov
o WP org: https://profiles.wordpress.org/igorkol
@IGORKO_HUB ON #WCVIE, APRIL 2015