Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25

35 views

Published on

Speech of Alexey Vasiliev, Software Engineer at Railsware, at Ruby Meditation #25 Kyiv 08.12.2018
Next conference - http://www.rubymeditation.com/

In this talk, Alexey will tell about the project in which was necessary to implement A/B testing and what came out of it in result

Announcements and conference materials https://www.fb.me/RubyMeditation
News https://twitter.com/RubyMeditation
Photos https://www.instagram.com/RubyMeditation
The stream of Ruby conferences (not just ours) https://t.me/RubyMeditation

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25

  1. 1. Road to A/B testing Brought to you by Oleksii Vasyliev 1
  2. 2. Oleksii Vasyliev Web and Mobile Developer (Ruby, Java, JavaScript, Objective-C, C/C++, Golang, Elixir/Erlang), DevOps Open-Source libs: PGTune, SQL Joins Visualizer, RWbox, Go-Kinesis, ElixirV8, WebP-ffi, Zopfli-ffi, MongodbLogger, SMTRails, SHTRails, ... Open-Source books: Setting up and scaling of PostgreSQL (Russian), Cooking Infrastructure by Chef Leading RWpod podcast about Ruby and JavaScript • • • • 2
  3. 3. What is A/B testing? 3
  4. 4. 4
  5. 5. First reaction 5
  6. 6. Rails solutions Split Vanity Field Test A/Bingo (dead) Bandit (dead) Absurdity (dead) SevenMinuteAbs (dead) • • • • • • • 6
  7. 7. Split Variants ab_test(:user_variations, 'new', 'old', 'strange') ab_test(:homepage_design, {'Old' => 18}, {'New' => 2}) Conversion ab_finished(:user_variations)7
  8. 8. Split Web interface gem 'split', require: 'split/dashboard' mount Split::Dashboard, at: 'split' 8
  9. 9. 9
  10. 10. Split Extensions Split::Export - Easily export A/B test data out of Split. Split::Analytics - Push test data to Google Analytics. Split::Mongoid - Store experiment data in mongoid (still uses redis). Split::Cacheable - Automatically create cache buckets per test. Split::Counters - Add counters per experiment and alternative. Split::Cli - A CLI to trigger Split A/B tests. • • • • • • 10
  11. 11. Demo for client 11
  12. 12. Small changes from client Support SPA applications (React, Vue.js, Marionette.js) Cover only some percentage of visitors to target an experiment I need determines when the experiment is shown: geo data, browser/mobile device, third-party cookies, query parameter, etc Running simple experiments (text/image variations, etc) without development and deployment Support Multivariate and Multi-Armed Bandit testing • • • • • 12
  13. 13. Multivariate tests (MVT) 13
  14. 14. Reaction 14
  15. 15. Ploblems I need better admin system for experiments (editor, settings, etc) I need collect info about customers (geo, device, etc) and do not forget about GDPR/Data Processing Addendum/etc It will create load on system Your customers not need this feature You as developer not need this code It is only need for marketing/business analytic team • • • • • • 15
  16. 16. What about PaaS? Optimizely ABTasty VWO Omniconvert Adobe Target etc • • • • • • 16
  17. 17. PaaS pricing Plans start at a minimum of $36,000 a year, for 200,000 visitors or more. https://www.quora.com/How-much-is-optimizely AB Tasty’s package starts at $1,000 per month https://reviews.financesonline.com/p/ab-tasty/#pricing Enterprise Pricing starts from $1,899/month https://reviews.financesonline.com/p/omniconvert/#pricing ” ” ” 17
  18. 18. Me 18
  19. 19. 19
  20. 20. ga('require', 'GTM-234234'); Google Optimize Integration ga('create', 'UA-234234-1', 'auto'); ga('send', 'pageview'); 20
  21. 21. Google Optimize Integration <style>.async-hide { opacity: 0 !important} </style> <script>(function(a,s,y,n,c,h,i,d,e){s.className+=' '+y;h.start=1*new Date; h.end=i=function(){s.className=s.className.replace(RegExp(' ?'+y),'')}; (a[n]=a[n]||[]).hide=h;setTimeout(function(){i();h.end=null},c);h.timeout=c; })(window,document.documentElement,'async-hide','dataLayer',4000, {'GTM-234234':true});</script> 21
  22. 22. 22
  23. 23. 23
  24. 24. 24
  25. 25. 25
  26. 26. 26
  27. 27. if (value === '2') { 'experiment_id_A' Google Optimize JS API function implementExperimentA(value) { $( document ).ready(function() { $('#my_element').text('Hello from variant 2'); }) } } gtag('event', 'optimize.callback', { name: , callback: implementExperimentA }); 27
  28. 28. A/B test results 28
  29. 29. 29
  30. 30. 30
  31. 31. 31
  32. 32. MVT setup 32
  33. 33. 33
  34. 34. 34
  35. 35. It is Free (with some limits) 35
  36. 36. Conclusion If you need PaaS for A/B testing - try Google Optimize Free version limits: 5 experiments, 10 personalizations, 16 MVT combinations. You can update to Optimize 360 if needed Ability to do experiments without development Works on top of a Google Analytics Integration with Google Tag Manager provide additional flexibility in settings • • • • • 36
  37. 37. <Thank You!> Questions? Contact information web: leopard.in.ua github: le0pard twitter: @leopard_me • • • 37

×