Java land What Users Want, A/B testing explained

What Users Want
A/B testing explained
Vladimir DejanovićJavaLand
14 March 2018
@VladimirD_42@JavaLandConf

Let’s Meet
@VladimirD_42
Vladimir Dejanović
@JavaLandConf

Let’s Meet
@VladimirD_42
Vladimir Dejanović and Others...
@JavaLandConf

Agenda

Agenda
@VladimirD_42
“Old way of working”
@JavaLandConf

Agenda
@VladimirD_42
A/B Testing
@JavaLandConf

Agenda
@VladimirD_42
Failing at A/B and Limitations
A/B Testing
@JavaLandConf

Agenda
@VladimirD_42
Failing at A/B and limitations
Questions
A/B Testing
@JavaLandConf

@VladimirD_42
Success vs Fail
@JavaLandConf

ref: https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAiMAAAAJDljNzE4YjZmLWJhNTAtNGM0My04Mzc1LTYxMDNlYWIzNDBhMw.jpg

ref: http://www.neuresourcegroup.com.au/brainwaves-for-leaders-blog//2012/08/09/hate-long-meetings-try-the-10-minute-standing-meeting/

ref: http://www.theinquirer.net/inquirer/news/2194702/nuance-announces-dragon-naturally-speaking-12-with-android-and-ios-support

ref: https://d1o50x50snmhul.cloudfront.net/wp-content/uploads/2016/07/05160722/f14aba_cropped-800x533.jpg

ref: https://d1o50x50snmhul.cloudfront.net/wp-content/uploads/2016/07/05160722/f14aba_cropped-800x533.jpg
Highest
Paid
Person
Opinion

ref: http://gogmat.com/blog

ref: https://macpeanut.wordpress.com/2015/08/12/crystal-balls/

ref: http://marketingland.com/view-mobile-retail-holidays-past-present-112419

ref: http://www.desktopanimewallpaper.com/wallpaper/Super-Saiyan/

@VladimirD_42
A/B Testing
@JavaLandConf

ref: http://www.truthmove.org/news-category/surveillance/

ref: http://www.rootsrundeep.com/hypothesis.html

ref: http://zehno.com/news-resources/the-more-you-know-ab-testing

ref: http://www.astekweb.com/the-web/what-is-ab-testing/

ref: http://www.pitstopmedia.com/sem/minimize-the-impact-of-website-testing-on-seo

ref: https://petovera.com/you-are-probably-getting-stupid-results-from-ab-testing-heres-why/

ref: http://www.footballhub.tk/champions-league-draw-round-16/

ref: https://www.graydon.co.uk/blog/data-driven-marketing-for-data-driven-marketers

ref: http://www.fanpop.com/clubs/mr-spock/images/10874060/title/mr-spock-photo

ref: https://s-media-cache-ak0.pinimg.com/originals/d3/14/d2/d314d211d6fe7f8967819858e4fa5333.jpg

Original

Base
Original

Base
Control
Original

Base
Control
Variant
Original

Base
Control
Variant Variant
Original

Base
Control
Variant Variant
Experiment
Original

Process

Visitors
Process

Visitors
Conversion
Process

A B

ref: https://vignette2.wikia.nocookie.net/breakingbad/images/9/9d/1x02_-_Coin_flip.jpg

B1 B1
A
A
A1 A1
B
B

ref: https://i.stack.imgur.com/sHtyh.png

ref: http://brightcove.vo.llnwd.net/d21/unsecured/media/1033249144001/1033249144001_2257652098001_video-still-for-video-2255602883001.jpg

ref: http://www.ru.nl/publish/pages/733019/mathematics-blackboard-article.jpg

Yes No
A
B
Total

Yes No
A A_Total
B
Total

Yes No
A A_Total
B B_Total
Total

Yes No
A A_Total
B B_Total
Total Total_Num

Yes No
A A_Yes A_Total
B B_Total
Total Total_Num

Yes No
A A_Yes A_No A_Total
B B_Total
Total Total_Num

Yes No
B B_Yes B_Total
Total Total_Num

Yes No
B B_Yes B_No B_Total
Total Total_Num

Yes No
Total Total_Yes Total_Num

Yes No
Total Total_Yes Total_No Total_Num

ref: http://anthonyrealestate.net/anthony3/wp-content/uploads/2011/12/crunching-numbers.jpg

Expectations
Ex_A_Yes
Ex_A_No
Ex_B_Yes
Ex_B_No

Expectations
Ex_A_Yes A_Total * Total_Yes / Total_Num
Ex_A_No
Ex_B_Yes
Ex_B_No

Expectations
Ex_A_No A_Total * Total_No / Total_Num
Ex_B_Yes
Ex_B_No

Expectations
Ex_A_No A_Total * Total_No / Total_Num
Ex_B_Yes B_Total * Total_Yes / Total_Num
Ex_B_No B_Total * Total_No / Total_Num

G-Test

G-Test
2 * Measured * log ( Measured /Expected )

G-Test
2 * ( A_Yes * log(A_Yes / Ex_A_Yes)
+ A_No * log(A_No / Ex_A_No)
+ B_Yes * log(B_Yes / Ex_B_Yes)
+ B_No * log(B_No / Ex_B_No) )

Chi-squared Probability Test

P = Chi_Square_Probability(1, G_Test)

P = Chi_Square_Probability(1, G_Test)
0 ≤ P≤ 1

Significant Difference

Sig_Diff = 1- P

Sig_Diff = 1- P
Sig_Diff = 0.95 => 95% chance

@VladimirD_42
Failing at A/B
and
Limitations
@JavaLandConf

Failing at A/B Testing

@VladimirD_42
Visitor != Saw Experiment
@JavaLandConf

@VladimirD_42
Visitor != Saw Experiment
Distribution for multiple experiments
@JavaLandConf

@VladimirD_42
Errors interfering with Experiment
@JavaLandConf

@VladimirD_42
Pausing Experiment
@JavaLandConf

@VladimirD_42
Not looking at whole process
@JavaLandConf

A/B
Experiment

A/B
Experiment
Issues

@VladimirD_42
Stopping experiment to fast
@JavaLandConf

Mon ThuWedTue Fri SunSat

@VladimirD_42
Understanding seasonality
@JavaLandConf

@VladimirD_42
Making to many changes in
one Experiment
@JavaLandConf

@VladimirD_42
Using Experiment 1 data
to prove “Experiment 2”
@JavaLandConf

Limitations

Limitations
@VladimirD_42
Number of Visitors
@JavaLandConf

Limitations
@VladimirD_42
No Big Bang
@JavaLandConf

@VladimirD_42
Thank You
@JavaLandConf

Questions
@VladimirD_42
@VladimirD_42
vladimir@itshark.xyz
@JavaLandConf

Java land What Users Want, A/B testing explained

Recommended

Recommended

More Related Content

More from Vladimir Dejanovic

More from Vladimir Dejanovic (14)

Recently uploaded

Recently uploaded (20)

Java land What Users Want, A/B testing explained