Vladimir Dejanović gave a presentation titled "What Users Want" at DevoxxPL on June 22, 2017. The document includes slides from the presentation discussing A/B testing methodology, including calculating expected values, the G-test statistic for measuring differences between measured and expected values, using the chi-squared probability test to calculate a P-value from the G-test, and determining statistical significance based on the P-value. Several references and examples are provided in the slides.

1. #DevoxxPL @VladimirD_42
What Users Want
Vladimir Dejanović
Krakow 22 June 2017

2. #DevoxxPL @VladimirD_42
Vladimir Dejanović
Let’s Meet

3. #DevoxxPL @VladimirD_42
ref: https://macpeanut.wordpress.com/2015/08/12/crystal-balls/

4. #DevoxxPL @VladimirD_42
ref: http://gogmat.com/blog

5. #DevoxxPL @VladimirD_42

6. #DevoxxPL @VladimirD_42
ref: https://gigaom.com/wp-content/uploads/sites/1/2011/02/2297179486_2089e1c8d1_z.jpg

7. #DevoxxPL @VladimirD_42
ref: http://marketingland.com/view-mobile-retail-holidays-past-present-112419

8. #DevoxxPL @VladimirD_42
ref: http://www.desktopanimewallpaper.com/wallpaper/Super-Saiyan/

9. #DevoxxPL @VladimirD_42
ref: http://www.truthmove.org/news-category/surveillance/

10. #DevoxxPL @VladimirD_42
ref: http://www.rootsrundeep.com/hypothesis.html

11. #DevoxxPL @VladimirD_42
ref: http://zehno.com/news-resources/the-more-you-know-ab-testing

12. #DevoxxPL @VladimirD_42
ref: http://www.astekweb.com/the-web/what-is-ab-testing/

13. #DevoxxPL @VladimirD_42
ref: http://www.pitstopmedia.com/sem/minimize-the-impact-of-website-testing-on-seo

14. #DevoxxPL @VladimirD_42
ref: https://petovera.com/you-are-probably-getting-stupid-results-from-ab-testing-heres-why/

15. #DevoxxPL @VladimirD_42
ref: http://www.footballhub.tk/champions-league-draw-round-16/

16. #DevoxxPL @VladimirD_42
ref: http://www.neuresourcegroup.com.au/brainwaves-for-leaders-blog//2012/08/09/hate-long-meetings-try-the-10-minute-standing-meeting/

17. #DevoxxPL @VladimirD_42
ref: http://www.theinquirer.net/inquirer/news/2194702/nuance-announces-dragon-naturally-speaking-12-with-android-and-ios-support

18. #DevoxxPL @VladimirD_42
ref: http://animalia.life/hippo.html

19. #DevoxxPL @VladimirD_42
ref: https://www.graydon.co.uk/blog/data-driven-marketing-for-data-driven-marketers

20. #DevoxxPL @VladimirD_42

21. #DevoxxPL @VladimirD_42
ref: http://www.fanpop.com/clubs/mr-spock/images/10874060/title/mr-spock-photo

22. #DevoxxPL @VladimirD_42
ref: https://s-media-cache-ak0.pinimg.com/originals/d3/14/d2/d314d211d6fe7f8967819858e4fa5333.jpg

23. #DevoxxPL @VladimirD_42

24. #DevoxxPL @VladimirD_42

25. #DevoxxPL @VladimirD_42

26. #DevoxxPL @VladimirD_42
A B

27. #DevoxxPL @VladimirD_42
A B

28. #DevoxxPL @VladimirD_42
A B

29. #DevoxxPL @VladimirD_42
ref: https://vignette2.wikia.nocookie.net/breakingbad/images/9/9d/1x02_-_Coin_flip.jpg

30. #DevoxxPL @VladimirD_42
A B

31. #DevoxxPL @VladimirD_42
B1 B1
A
A
A1 A1
B
B

32. #DevoxxPL @VladimirD_42
ref: https://i.stack.imgur.com/sHtyh.png

33. #DevoxxPL @VladimirD_42
ref: http://brightcove.vo.llnwd.net/d21/unsecured/media/1033249144001/1033249144001_2257652098001_video-still-for-video-2255602883001.jpg

34. #DevoxxPL @VladimirD_42
ref: http://www.ru.nl/publish/pages/733019/mathematics-blackboard-article.jpg

35. #DevoxxPL @VladimirD_42
Yes No
A A_Total
B
Total

36. #DevoxxPL @VladimirD_42
Yes No
A A_Total
B B_Total
Total

37. #DevoxxPL @VladimirD_42
Yes No
A A_Total
B B_Total
Total Total_Num

38. #DevoxxPL @VladimirD_42
Yes No
A A_Yes A_Total
B B_Total
Total Total_Num

39. #DevoxxPL @VladimirD_42
Yes No
A A_Yes A_No A_Total
B B_Total
Total Total_Num

40. #DevoxxPL @VladimirD_42
Yes No
A A_Yes A_No A_Total
B B_Yes B_Total
Total Total_Num

41. #DevoxxPL @VladimirD_42
Yes No
A A_Yes A_No A_Total
B B_Yes B_No B_Total
Total Total_Num

42. #DevoxxPL @VladimirD_42
Yes No
A A_Yes A_No A_Total
B B_Yes B_No B_Total
Total Total_Yes Total_Num

43. #DevoxxPL @VladimirD_42
Yes No
A A_Yes A_No A_Total
B B_Yes B_No B_Total
Total Total_Yes Total_No Total_Num

44. #DevoxxPL @VladimirD_42
ref: http://anthonyrealestate.net/anthony3/wp-content/uploads/2011/12/crunching-numbers.jpg

45. #DevoxxPL @VladimirD_42
Ex_A_Yes
Ex_A_No
Ex_B_Yes
Ex_B_No
Expectations

46. #DevoxxPL @VladimirD_42
Ex_A_Yes A_Total * Total_Yes / Total_Num
Ex_A_No
Ex_B_Yes
Ex_B_No
Expectations

47. #DevoxxPL @VladimirD_42
Ex_A_Yes A_Total * Total_Yes / Total_Num
Ex_A_No A_Total * Total_No / Total_Num
Ex_B_Yes
Ex_B_No
Expectations

48. #DevoxxPL @VladimirD_42
Ex_A_Yes A_Total * Total_Yes / Total_Num
Ex_A_No A_Total * Total_No / Total_Num
Ex_B_Yes B_Total * Total_Yes / Total_Num
Ex_B_No B_Total * Total_No / Total_Num
Expectations

49. #DevoxxPL @VladimirD_42
G-Test

50. #DevoxxPL @VladimirD_42
G-Test
2 * Measured * log ( Measured /Expected )

51. #DevoxxPL @VladimirD_42
G-Test
2 * ( A_Yes * log(A_Yes / Ex_A_Yes)
+ A_No * log(A_No / Ex_A_No)
+ B_Yes * log(B_Yes / Ex_B_Yes)
+ B_No * log(B_No / Ex_B_No) )

52. #DevoxxPL @VladimirD_42
Chi-squared Probability Test

53. #DevoxxPL @VladimirD_42
Chi-squared Probability Test
P = Chi_Square_Probability(1, G_Test)

54. #DevoxxPL @VladimirD_42
Chi-squared Probability Test
P = Chi_Square_Probability(1, G_Test)
0 ≤ P≤ 1

55. #DevoxxPL @VladimirD_42
Significant Difference

56. #DevoxxPL @VladimirD_42
Significant Difference
Sig_Diff = 1- P

57. #DevoxxPL @VladimirD_42
Significant Difference
Sig_Diff = 1- P
Sig_Diff = 0.95 => 95% chance

58. #DevoxxPL @VladimirD_42
Questions
@VladimirD_42
ed.wong.iv@gmail.com

59. #DevoxxPL @VladimirD_42
Thank You