1. Road to A/B testing
Brought to you by Oleksii Vasyliev
1

2. Oleksii Vasyliev
Web and Mobile Developer (Ruby, Java, JavaScript, Objective-C, C/C++,
Golang, Elixir/Erlang), DevOps
Open-Source libs: PGTune, SQL Joins Visualizer, RWbox, Go-Kinesis, ElixirV8,
WebP-ffi, Zopfli-ffi, MongodbLogger, SMTRails, SHTRails, ...
Open-Source books: Setting up and scaling of PostgreSQL (Russian),
Cooking Infrastructure by Chef
Leading RWpod podcast about Ruby and JavaScript
•
•
•
•
2

3. What is A/B testing?
3

4. 4

5. First reaction
5

6. Rails solutions
Split
Vanity
Field Test
A/Bingo (dead)
Bandit (dead)
Absurdity (dead)
SevenMinuteAbs (dead)
•
•
•
•
•
•
•
6

7. Split
Variants
ab_test(:user_variations, 'new', 'old', 'strange')
ab_test(:homepage_design, {'Old' => 18}, {'New' => 2})
Conversion
ab_finished(:user_variations)7

8. Split Web interface
gem 'split', require: 'split/dashboard'
mount Split::Dashboard, at: 'split'
8

9. 9

10. Split Extensions
Split::Export - Easily export A/B test data out of Split.
Split::Analytics - Push test data to Google Analytics.
Split::Mongoid - Store experiment data in mongoid (still uses redis).
Split::Cacheable - Automatically create cache buckets per test.
Split::Counters - Add counters per experiment and alternative.
Split::Cli - A CLI to trigger Split A/B tests.
•
•
•
•
•
•
10

11. Demo for client
11

12. Small changes from client
Support SPA applications (React, Vue.js, Marionette.js)
Cover only some percentage of visitors to target an experiment
I need determines when the experiment is shown: geo data, browser/mobile
device, third-party cookies, query parameter, etc
Running simple experiments (text/image variations, etc) without
development and deployment
Support Multivariate and Multi-Armed Bandit testing
•
•
•
•
•
12

13. Multivariate tests (MVT)
13

14. Reaction
14

15. Ploblems
I need better admin system for experiments (editor, settings, etc)
I need collect info about customers (geo, device, etc) and do not forget
about GDPR/Data Processing Addendum/etc
It will create load on system
Your customers not need this feature
You as developer not need this code
It is only need for marketing/business analytic team
•
•
•
•
•
•
15

16. What about PaaS?
Optimizely
ABTasty
VWO
Omniconvert
Adobe Target
etc
•
•
•
•
•
•
16

17. PaaS pricing
Plans start at a minimum of $36,000 a year, for 200,000 visitors or more.
https://www.quora.com/How-much-is-optimizely
AB Tasty’s package starts at $1,000 per month
https://reviews.financesonline.com/p/ab-tasty/#pricing
Enterprise Pricing starts from $1,899/month
https://reviews.financesonline.com/p/omniconvert/#pricing
”
”
” 17

18. Me
18

19. 19

20. ga('require', 'GTM-234234');
Google Optimize Integration
ga('create', 'UA-234234-1', 'auto');
ga('send', 'pageview');
20

21. Google Optimize Integration
<style>.async-hide { opacity: 0 !important} </style>
<script>(function(a,s,y,n,c,h,i,d,e){s.className+=' '+y;h.start=1*new Date;
h.end=i=function(){s.className=s.className.replace(RegExp(' ?'+y),'')};
(a[n]=a[n]||[]).hide=h;setTimeout(function(){i();h.end=null},c);h.timeout=c;
})(window,document.documentElement,'async-hide','dataLayer',4000,
{'GTM-234234':true});</script>
21

22. 22

23. 23

24. 24

25. 25

26. 26

27. if (value === '2') {
'experiment_id_A'
Google Optimize JS API
function implementExperimentA(value) {
$( document ).ready(function() {
$('#my_element').text('Hello from variant 2');
})
}
}
gtag('event', 'optimize.callback', {
name: ,
callback: implementExperimentA
}); 27

28. A/B test results
28

29. 29

30. 30

31. 31

32. MVT setup
32

33. 33

34. 34

35. It is Free (with some limits)
35

36. Conclusion
If you need PaaS for A/B testing - try Google Optimize
Free version limits: 5 experiments, 10 personalizations, 16 MVT
combinations. You can update to Optimize 360 if needed
Ability to do experiments without development
Works on top of a Google Analytics
Integration with Google Tag Manager provide additional flexibility in settings
•
•
•
•
•
36

37. <Thank You!> Questions?
Contact information
web: leopard.in.ua
github: le0pard
twitter: @leopard_me
•
•
•
37