Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Road to A/B testing
Brought to you by Oleksii Vasyliev
1
Oleksii Vasyliev
Web and Mobile Developer (Ruby, Java, JavaScript, Objective-C, C/C++,
Golang, Elixir/Erlang), DevOps
Open...
What is A/B testing?
3
4
First reaction
5
Rails solutions
Split
Vanity
Field Test
A/Bingo (dead)
Bandit (dead)
Absurdity (dead)
SevenMinuteAbs (dead)
•
•
•
•
•
•
•
6
Split
Variants
ab_test(:user_variations, 'new', 'old', 'strange')
ab_test(:homepage_design, {'Old' => 18}, {'New' => 2})
C...
Split Web interface
gem 'split', require: 'split/dashboard'
mount Split::Dashboard, at: 'split'
8
9
Split Extensions
Split::Export - Easily export A/B test data out of Split.
Split::Analytics - Push test data to Google Ana...
Demo for client
11
Small changes from client
Support SPA applications (React, Vue.js, Marionette.js)
Cover only some percentage of visitors t...
Multivariate tests (MVT)
13
Reaction
14
Ploblems
I need better admin system for experiments (editor, settings, etc)
I need collect info about customers (geo, devi...
What about PaaS?
Optimizely
ABTasty
VWO
Omniconvert
Adobe Target
etc
•
•
•
•
•
•
16
PaaS pricing
Plans start at a minimum of $36,000 a year, for 200,000 visitors or more.
https://www.quora.com/How-much-is-o...
Me
18
19
ga('require', 'GTM-234234');
Google Optimize Integration
ga('create', 'UA-234234-1', 'auto');
ga('send', 'pageview');
20
Google Optimize Integration
<style>.async-hide { opacity: 0 !important} </style>
<script>(function(a,s,y,n,c,h,i,d,e){s.cl...
22
23
24
25
26
if (value === '2') {
'experiment_id_A'
Google Optimize JS API
function implementExperimentA(value) {
$( document ).ready(f...
A/B test results
28
29
30
31
MVT setup
32
33
34
It is Free (with some limits)
35
Conclusion
If you need PaaS for A/B testing - try Google Optimize
Free version limits: 5 experiments, 10 personalizations,...
<Thank You!> Questions?
Contact information
web: leopard.in.ua
github: le0pard
twitter: @leopard_me
•
•
•
37
Upcoming SlideShare
Loading in …5
×

of

Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 1 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 2 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 3 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 4 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 5 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 6 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 7 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 8 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 9 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 10 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 11 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 12 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 13 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 14 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 15 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 16 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 17 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 18 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 19 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 20 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 21 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 22 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 23 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 24 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 25 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 26 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 27 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 28 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 29 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 30 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 31 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 32 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 33 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 34 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 35 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 36 Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25 Slide 37
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25

Download to read offline

Speech of Alexey Vasiliev, Software Engineer at Railsware, at Ruby Meditation #25 Kyiv 08.12.2018
Next conference - http://www.rubymeditation.com/

In this talk, Alexey will tell about the project in which was necessary to implement A/B testing and what came out of it in result

Announcements and conference materials https://www.fb.me/RubyMeditation
News https://twitter.com/RubyMeditation
Photos https://www.instagram.com/RubyMeditation
The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25

  1. 1. Road to A/B testing Brought to you by Oleksii Vasyliev 1
  2. 2. Oleksii Vasyliev Web and Mobile Developer (Ruby, Java, JavaScript, Objective-C, C/C++, Golang, Elixir/Erlang), DevOps Open-Source libs: PGTune, SQL Joins Visualizer, RWbox, Go-Kinesis, ElixirV8, WebP-ffi, Zopfli-ffi, MongodbLogger, SMTRails, SHTRails, ... Open-Source books: Setting up and scaling of PostgreSQL (Russian), Cooking Infrastructure by Chef Leading RWpod podcast about Ruby and JavaScript • • • • 2
  3. 3. What is A/B testing? 3
  4. 4. 4
  5. 5. First reaction 5
  6. 6. Rails solutions Split Vanity Field Test A/Bingo (dead) Bandit (dead) Absurdity (dead) SevenMinuteAbs (dead) • • • • • • • 6
  7. 7. Split Variants ab_test(:user_variations, 'new', 'old', 'strange') ab_test(:homepage_design, {'Old' => 18}, {'New' => 2}) Conversion ab_finished(:user_variations)7
  8. 8. Split Web interface gem 'split', require: 'split/dashboard' mount Split::Dashboard, at: 'split' 8
  9. 9. 9
  10. 10. Split Extensions Split::Export - Easily export A/B test data out of Split. Split::Analytics - Push test data to Google Analytics. Split::Mongoid - Store experiment data in mongoid (still uses redis). Split::Cacheable - Automatically create cache buckets per test. Split::Counters - Add counters per experiment and alternative. Split::Cli - A CLI to trigger Split A/B tests. • • • • • • 10
  11. 11. Demo for client 11
  12. 12. Small changes from client Support SPA applications (React, Vue.js, Marionette.js) Cover only some percentage of visitors to target an experiment I need determines when the experiment is shown: geo data, browser/mobile device, third-party cookies, query parameter, etc Running simple experiments (text/image variations, etc) without development and deployment Support Multivariate and Multi-Armed Bandit testing • • • • • 12
  13. 13. Multivariate tests (MVT) 13
  14. 14. Reaction 14
  15. 15. Ploblems I need better admin system for experiments (editor, settings, etc) I need collect info about customers (geo, device, etc) and do not forget about GDPR/Data Processing Addendum/etc It will create load on system Your customers not need this feature You as developer not need this code It is only need for marketing/business analytic team • • • • • • 15
  16. 16. What about PaaS? Optimizely ABTasty VWO Omniconvert Adobe Target etc • • • • • • 16
  17. 17. PaaS pricing Plans start at a minimum of $36,000 a year, for 200,000 visitors or more. https://www.quora.com/How-much-is-optimizely AB Tasty’s package starts at $1,000 per month https://reviews.financesonline.com/p/ab-tasty/#pricing Enterprise Pricing starts from $1,899/month https://reviews.financesonline.com/p/omniconvert/#pricing ” ” ” 17
  18. 18. Me 18
  19. 19. 19
  20. 20. ga('require', 'GTM-234234'); Google Optimize Integration ga('create', 'UA-234234-1', 'auto'); ga('send', 'pageview'); 20
  21. 21. Google Optimize Integration <style>.async-hide { opacity: 0 !important} </style> <script>(function(a,s,y,n,c,h,i,d,e){s.className+=' '+y;h.start=1*new Date; h.end=i=function(){s.className=s.className.replace(RegExp(' ?'+y),'')}; (a[n]=a[n]||[]).hide=h;setTimeout(function(){i();h.end=null},c);h.timeout=c; })(window,document.documentElement,'async-hide','dataLayer',4000, {'GTM-234234':true});</script> 21
  22. 22. 22
  23. 23. 23
  24. 24. 24
  25. 25. 25
  26. 26. 26
  27. 27. if (value === '2') { 'experiment_id_A' Google Optimize JS API function implementExperimentA(value) { $( document ).ready(function() { $('#my_element').text('Hello from variant 2'); }) } } gtag('event', 'optimize.callback', { name: , callback: implementExperimentA }); 27
  28. 28. A/B test results 28
  29. 29. 29
  30. 30. 30
  31. 31. 31
  32. 32. MVT setup 32
  33. 33. 33
  34. 34. 34
  35. 35. It is Free (with some limits) 35
  36. 36. Conclusion If you need PaaS for A/B testing - try Google Optimize Free version limits: 5 experiments, 10 personalizations, 16 MVT combinations. You can update to Optimize 360 if needed Ability to do experiments without development Works on top of a Google Analytics Integration with Google Tag Manager provide additional flexibility in settings • • • • • 36
  37. 37. <Thank You!> Questions? Contact information web: leopard.in.ua github: le0pard twitter: @leopard_me • • • 37

Speech of Alexey Vasiliev, Software Engineer at Railsware, at Ruby Meditation #25 Kyiv 08.12.2018 Next conference - http://www.rubymeditation.com/ In this talk, Alexey will tell about the project in which was necessary to implement A/B testing and what came out of it in result Announcements and conference materials https://www.fb.me/RubyMeditation News https://twitter.com/RubyMeditation Photos https://www.instagram.com/RubyMeditation The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Views

Total views

539

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

4

Shares

0

Comments

0

Likes

0

×