What Users Want, A/B testing explained [CodeteCon 2018]

#CodeteCON @VladimirD_42
What Users Want
A/B testing explained
Vladimir Dejanović
11 October 2018

@VladimirD_42
Vladimir Dejanović
Let’s Meet
@VladimirD_42
vladimir@itshark.xyz
http://itshark.xyz
#CodeteCON

@VladimirD_42
Vladimir Dejanović
Let’s Meet
http://itshark.xyz
#CodeteCON

@VladimirD_42
Vladimir Dejanović
Let’s Meet
#CodeteCON

@VladimirD_42
Vladimir Dejanović
Let’s Meet
and Others...
#CodeteCON

@VladimirD_42#CodeteCON
Agenda

Agenda
Success vs Fail

Agenda
A/B Testing
Success vs Fail

Agenda
A/B Testing
Failing at A/B and limitations
Success vs Fail

Agenda
A/B Testing
Questions
Failing at A/B and limitations
Success vs Fail

Success vs Fail

ref: https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAiMAAAAJDljNzE4YjZmLWJhNTAtNGM0My04Mzc1LTYxMDNlYWIzNDBhMw.jpg

ref: http://www.neuresourcegroup.com.au/brainwaves-for-leaders-blog//2012/08/09/hate-long-meetings-try-the-10-minute-standing-meeting/

ref: http://www.theinquirer.net/inquirer/news/2194702/nuance-announces-dragon-naturally-speaking-12-with-android-and-ios-support

ref: https://d1o50x50snmhul.cloudfront.net/wp-content/uploads/2016/07/05160722/f14aba_cropped-800x533.jpg

ref: https://d1o50x50snmhul.cloudfront.net/wp-content/uploads/2016/07/05160722/f14aba_cropped-800x533.jpg
Highest
Paid
Person
Opinion

ref: http://gogmat.com/blog

ref: https://macpeanut.wordpress.com/2015/08/12/crystal-balls/

ref: http://marketingland.com/view-mobile-retail-holidays-past-present-112419

ref: http://www.desktopanimewallpaper.com/wallpaper/Super-Saiyan/

A/B Testing

ref: http://www.truthmove.org/news-category/surveillance/

ref: http://www.rootsrundeep.com/hypothesis.html

ref: http://zehno.com/news-resources/the-more-you-know-ab-testing

ref: http://www.astekweb.com/the-web/what-is-ab-testing/

ref: http://www.pitstopmedia.com/sem/minimize-the-impact-of-website-testing-on-seo

ref: https://petovera.com/you-are-probably-getting-stupid-results-from-ab-testing-heres-why/

ref: http://www.footballhub.tk/champions-league-draw-round-16/

ref: https://www.graydon.co.uk/blog/data-driven-marketing-for-data-driven-marketers

ref: http://www.fanpop.com/clubs/mr-spock/images/10874060/title/mr-spock-photo

ref: https://s-media-cache-ak0.pinimg.com/originals/d3/14/d2/d314d211d6fe7f8967819858e4fa5333.jpg

Original

Base
Original

Base
Control
Original

Base
Control
Variant
Original

Base
Control
Variant Variant
Original

Base
Control
Variant Variant
Experiment
Original

Process

Visitors
Process

Visitors
Conversion
Process

ref: https://vignette2.wikia.nocookie.net/breakingbad/images/9/9d/1x02_-_Coin_flip.jpg

B1 B1
A
A
A1 A1
B
B

ref: https://i.stack.imgur.com/sHtyh.png

ref: http://brightcove.vo.llnwd.net/d21/unsecured/media/1033249144001/1033249144001_2257652098001_video-still-for-video-2255602883001.jpg

ref: https://www.geek-vintage.com/wp-content/uploads/invention-gaston.jpg

ref: http://www.ru.nl/publish/pages/733019/mathematics-blackboard-article.jpg

Yes No
A
B
Total

Yes No
A A_Total
B
Total

Yes No
A A_Total
B B_Total
Total

Yes No
A A_Total
B B_Total
Total Total_Num

Yes No
A A_Yes A_Total
B B_Total
Total Total_Num

Yes No
A A_Yes A_No A_Total
B B_Total
Total Total_Num

Yes No
B B_Yes B_Total
Total Total_Num

Yes No
B B_Yes B_No B_Total
Total Total_Num

Yes No
Total Total_Yes Total_Num

Yes No
Total Total_Yes Total_No Total_Num

ref: http://anthonyrealestate.net/anthony3/wp-content/uploads/2011/12/crunching-numbers.jpg

Expectations
Ex_A_Yes
Ex_A_No
Ex_B_Yes
Ex_B_No

Expectations
Ex_A_Yes A_Total * Total_Yes / Total_Num
Ex_A_No
Ex_B_Yes
Ex_B_No

Expectations
Ex_A_No A_Total * Total_No / Total_Num
Ex_B_Yes
Ex_B_No

Expectations
Ex_A_No A_Total * Total_No / Total_Num
Ex_B_Yes B_Total * Total_Yes / Total_Num
Ex_B_No B_Total * Total_No / Total_Num

G-Test

G-Test
2 * Measured * log ( Measured /Expected )

G-Test
2 * ( A_Yes * log(A_Yes / Ex_A_Yes)
+ A_No * log(A_No / Ex_A_No)
+ B_Yes * log(B_Yes / Ex_B_Yes)
+ B_No * log(B_No / Ex_B_No) )

Chi-squared Probability Test

P = Chi_Square_Probability(1, G_Test)

P = Chi_Square_Probability(1, G_Test)
0 ≤ P≤ 1

Significant Difference

Sig_Diff = 1- P

Sig_Diff = 1- P
Sig_Diff = 0.95 => 95% chance

Failing at A/B
and
Limitations

Failing at A/B Testing

Visitor != Saw Experiment

Visitor != Saw Experiment
Distribution for multiple experiments

Errors interfering with Experiment

Not looking at whole process

A/B
Experiment

A/B
Experiment
Issues

Stopping experiment to fast

Mon ThuWedTue Fri SunSat

Understanding seasonality

ref: http://aws-cf.caradisiac.com/prod/uploads/200401/ziriako_1073088347_gasapo1f.gif

Using Experiment 1 data
to prove “Experiment 2”

Limitations

ref: http://www.tashakheiriddin.com/wp-content/uploads/2016/12/chicken-or-egg2.jpg

ref: https://media3.s-nbcnews.com/j/newscms/2018_23/2455046/180605-big-bang-illustration-al-1141_c426a54e8d4f6d67f0b33d1abb671d40.fit-2000w.jpg

Thank You

@VladimirD_42
Questions
@VladimirD_42
vladimir@itshark.xyz
http://itshark.xyz
#CodeteCON

What Users Want, A/B testing explained [CodeteCon 2018]

More Related Content

More from Vladimir Dejanovic

Recently uploaded

What Users Want, A/B testing explained [CodeteCon 2018]