Your SlideShare is downloading. ×
0
Validated Learning at TechnoWeb
A. Oertl, M. Heiss, B. Laenger & B. Kavsek | Corporate Technology | Sep. 2013

siemens.com...
Have you read this book?

WARNING:

Each day you delay reading this book
you risk wasting money.

Source: http://www.amazo...
Reduce Cycle Time
Evade the worst impact on productivity: Do not build something nobody wants

Solution:
• Get immediate f...
Meaningful progress:
Validated Learning
Defining success:
• Success is improved customer behavior
• Success is measured ei...
Agenda

Page 5

September 2013

Siemens CT TIM CEE

Unrestricted © Siemens AG 2013. All rights reserved
Corporate Problem Solving via TechnoWeb:
Ask an Urgent Request and get answers from peers
Urgent Requests are distributed ...
New is not always better
Requirement: Urgent Request notifications had to be changed to fit corporate design guidelines

W...
Releasing new features without validated learning
is like being in the dark
Decisions are often made using one‟s own best ...
Agenda

Page 9

September 2013

Siemens CT TIM CEE

Unrestricted © Siemens AG 2013. All rights reserved
Statistical Evaluation by Engineers
without Specialized Statistical Knowledge

Page 10

September 2013

Siemens CT TIM CEE...
Split-Test:
Preparing to prove assumptions
Split Test: Define a hypothesis with metric and expected value
Hypothesis

Urge...
Results of 323.560 Urgent Request notifications

Histogram: Conversion rate ratio
convnew/convold
5

12
Count of Urgent Re...
Decreasing click-through rate ratio
with increasing Business Impact

Visibility of Business Impact Level: Impact on perfor...
Split-Tests in enterprises face
lower statistical significance
Decisions had to be made concerning raw data processing:
Lo...
Statistical Evaluation
by Statisticians

Page 15

September 2013

Siemens CT TIM CEE

Unrestricted © Siemens AG 2013. All ...
Techno Web Split Analysis:
Old versus new Template for Urgent Requests

Approach for sample selection:

0%

50%

100%

Urg...
Statistical Questions

• Statistical Question to be answered by the analysis:
• Is there a difference in the number of res...
Sample Characteristics
Dependency within 1 observation?
• Are we considering paired or unpaired samples?
- Paired sample m...
Sample Characteristics
Independency between observations?
The problem is that for most statistical tests, values between o...
Selection of Test Method
Comparison of means:
Is the mean response significantly different in the new template compared to...
Check of premises
Before applying a hypothesis test, the differences (v0-v1 and c0-c1) have to be
tested on normal distrib...
Check of premises
Symmetrical Distribution of differences v0-v1 and c0-c1:

Page 22

September 2013

Siemens CT TIM CEE
Hypothesis Test:
Principle of the Wilcoxon rank sum test
Wilcoxon rank sum test (U-test for paired samples):
Example for n...
Test Results
Results of Wilcoxon rank sum test:
H0

p-value

v0= v1

1
9.076e-07

c0= c1

0.4616

c0>

c1

0.7718

c0< c1
...
Plots: response for old versus new template
.
Views in old (black)
versus new (red)
template

Comments in old
(black) vers...
Variable for comparison of old and new template

Click-through ratio and conversion ratio: Problem of exclusion of zero va...
Variable for comparison of old and new template
Using differences v1-v0 and c1-c0 instead of quotients v1/v0 and c1/c0
 z...
New First-Timers
• Considering only subgroup receiving the new template:
Is there a correlation between the number of “new...
New First-Timers
• Considering only subgroup receiving the new template:
Is there a correlation between the number of “new...
Sample Characteristics
Improvement suggestion for novel split test

Proposition of sample selection for next split test:
•...
Results and Recommendations
The following results were obtained:
•

No significant change in number of comments in new ver...
Agenda

Page 32

September 2013

Siemens CT TIM CEE

Unrestricted © Siemens AG 2013. All rights reserved
Learning:
The new template has three usability problems
• The overall performance of the new template is inferior to the o...
Moving key elements to the side
decreased their effectiveness

Page 34

September 2013

Siemens CT TIM CEE

Unrestricted ©...
Conclusion

Specific

General

• Split-testing a new feature is worth the time
and effort.

• Features that do not positiv...
Upcoming SlideShare
Loading in...5
×

TechnoWeb Split Test in the context of validated learning

690

Published on

This talk was given at the i-know 2013 and the IEEE TMC Chapter CE Meeting in November 2013. Authors are Andreas Oertl (frist author), Michael Heiss, Bettina Laenger, Barbara Kavsek. This time a more detailed presentation about a split test for the urgent request notification within Siemens TechnoWeb (and it's statistical significance analysis)

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
690
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "TechnoWeb Split Test in the context of validated learning"

  1. 1. Validated Learning at TechnoWeb A. Oertl, M. Heiss, B. Laenger & B. Kavsek | Corporate Technology | Sep. 2013 siemens.com/answers Unrestricted © Siemens AG 2013. All rights reserved
  2. 2. Have you read this book? WARNING: Each day you delay reading this book you risk wasting money. Source: http://www.amazon.de/The-Lean-Startup-Entrepreneurs-Continuous/dp/0307887898 Page 2 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  3. 3. Reduce Cycle Time Evade the worst impact on productivity: Do not build something nobody wants Solution: • Get immediate feedback from customers Source: http://www.betterthanpants.com/baby-mop.html Page 3 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  4. 4. Meaningful progress: Validated Learning Defining success: • Success is improved customer behavior • Success is measured either by generally applicable metrics, or metrics tailored to a specific situation. Metric examples: • Value hypothesis • Retention rate (generic): How many customers return within a set time period? • UR-conversion rate (custom): How many per mill of the notified users respond to the question? • Growth hypothesis • Cohort based (generic): Separate behavior analysis of independent user groups (e.g. monthly new users). • Invitation rate (generic): The willingness of users to invite their personal contacts to the same service. • The results are used to decide if the change in the feature has positive, negative or no effects on consumer behavior. • This way, learning immediately delivers business relevant insights. Page 4 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  5. 5. Agenda Page 5 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  6. 6. Corporate Problem Solving via TechnoWeb: Ask an Urgent Request and get answers from peers Urgent Requests are distributed per email to the relevant target group (target messaging) Headline of the Urgent Request Business Impact (estimated by sender) Many replies on average 7 replies, first within 35min. 90% get help Name and optional photo of the sender Page 6 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  7. 7. New is not always better Requirement: Urgent Request notifications had to be changed to fit corporate design guidelines Which solution is better? Page 7 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  8. 8. Releasing new features without validated learning is like being in the dark Decisions are often made using one‟s own best judgment, ignoring customer needs. Common approach: Validated Learning • The automatic conclusion: the new feature is “obviously better” than the old one, and the time and money for the improvement were well spent. A meaningful conclusion can only be drawn after this question is answered: Does the change positively influence customer behavior? • Urgent Requests are the most important functionality of TechnoWeb • The e-mail notification invites users to give answers • Therefore, the effectiveness of the notification is mission critical for the success of TechnoWeb.  It is imperative to measure customer response to the new template. Page 8 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  9. 9. Agenda Page 9 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  10. 10. Statistical Evaluation by Engineers without Specialized Statistical Knowledge Page 10 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  11. 11. Split-Test: Preparing to prove assumptions Split Test: Define a hypothesis with metric and expected value Hypothesis Urgent Request number i The new template outperforms the old template in: Approx. 50% of the users receive the old template Approx. 50% of the users receive the new template • Click-through rate SPLIT • Conversion rate Ei, old Ei, new Vi, old Vi, new Ci, old Ci, new The introduced metrics are: • Click-through rate ratio • Conversion rate ratio • i…Urgent Request number • Ei…number of sent notifications • Vi…number of views • old…old template • Ci…number of comments • new…new template Page 11 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  12. 12. Results of 323.560 Urgent Request notifications Histogram: Conversion rate ratio convnew/convold 5 12 Count of Urgent Requests with corresponding ratio Count of Urgent Requests with corresponding ratio Histogram: Click-through rate ratio ctrnew/ctrold 10 8 6 4 2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 Click-through rate ratio 4 3 2 1 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 1,1 1,2 1,3 1,4 1,5 1,6 1,7 1,8 1,9 2 >2 Conversion rate ratio • The click-through and conversion rate ratios compare the relative success (relative to the number of notifications sent) of the old and new templates. A value <1 means that the performance of the new template is inferior to the old template. A value of 1 signifies no change, whereas a value >1 indicates a better performing new template. • For the click-through rate ratio, all 61 Urgent Requests are considered. For the conversion rate ratio, only 32 Urgent Requests have sufficient data to be used. Page 12 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  13. 13. Decreasing click-through rate ratio with increasing Business Impact Visibility of Business Impact Level: Impact on performance • The Business Impact Level assigns a monetary value to the problem statement of the Urgent Request • A value <1 means that the performance of the new template is inferior to the old template • The monetary value is displayed less prominently in the new template. Instead of assuming an impact, we measure it: Average Click-through rate ratio Average Conversion rate ratio 1.2 1.0 0.8 0.6 0.4 0.2 0.0 €1,000 €10,000 €50,000 €250,000 €1,000,000 Average Conversion rate ratio Average Click-through ratio 1.2 1.0 0.8 0.6 0.4 0.2 0.0 €1,000 €10,000 €50,000 €250,000 €1,000,000 Business Impact Business Impact Page 13 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  14. 14. Split-Tests in enterprises face lower statistical significance Decisions had to be made concerning raw data processing: Low comment count Statistical significance • Problem • Problem Even though 323.560 notifications were evaluated, it‟s in the nature of the application that the absolute number of comments are low. In some cases +/- 1 comment can significantly influence the result. • Solution • Disregard multiple comments by the same user (e.g. follow-up comments). Some Urgent Requests, with 25.000 email notifications, are statistically significant. Others send only a couple of hundred emails. • Solution Discard all data sets where no significant data (views or comments) has been recorded from either old or new template. • Disregard all activity by the author of the Urgent Request. • Discard all data sets where there are no comments from both the old and new template. This comes at the cost of less data to work with, but the remaining data is much more trustworthy. Page 14 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  15. 15. Statistical Evaluation by Statisticians Page 15 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  16. 16. Techno Web Split Analysis: Old versus new Template for Urgent Requests Approach for sample selection: 0% 50% 100% Urgent Request 1 NEW OLD Urgent Request t1 Before first-time OLD Urgent Request t2 UR 1 .. t1 Before UR 1 .. t2 Receivers of NEW template will always receive NEW further on. Page 16 September 2013 Siemens CT TIM CEE OLD If >50% have already received new template  more „NEW“ than „OLD“
  17. 17. Statistical Questions • Statistical Question to be answered by the analysis: • Is there a difference in the number of responses (views, comments) of the old versus new template? • Do first-time users of the new template behave differently from users that received the new template before? • Requested for future analyses: • Is there one representative number for the extent of this difference, considering all urgent requests? Page 17 September 2013 Siemens CT TIM CEE
  18. 18. Sample Characteristics Dependency within 1 observation? • Are we considering paired or unpaired samples? - Paired sample means that 2 characteristics of one observation are dependent - We want to compare responses (views, comments) to the same urgent request for old versus new template. - Thus, we have to consider pairs of responses and investigate the difference between response ratios for each urgent request. - Example: click-through ratio old click-through ratio new Urgent request 1 0.01 0.03 Urgent request 2 0.03 0.05 Urgent request 3 0.07 0.01 Urgent request 4 0.05 0.07 Assuming independent samples  assuming equal mean in old and new template. BUT: In reality: ctrold < ctrnew in ¾ of requests!  We assume dependent samples  paired test Page 18 September 2013 Siemens CT TIM CEE
  19. 19. Sample Characteristics Independency between observations? The problem is that for most statistical tests, values between observations of the sample (i.e. different urgent requests) have to be independent. We know that the same person gets several urgent requests, however, it is assumed that the response behavior (to click on the notification link) is independent for different topics. Thus we can assume independence of the different urgent requests. click-through ratio old Urgent request 1 Page 19 September 2013 0.01 0.03 Urgent request 2 0.03 0.05 Urgent request 3 0.07 0.01 Urgent request 4 independent click-through ratio new 0.05 0.07 Siemens CT TIM CEE dependent
  20. 20. Selection of Test Method Comparison of means: Is the mean response significantly different in the new template compared to the old template? • t-Test for paired samples Premises: - 2 paired samples (xi,yi) with expectation values 1 and 2 - Differences di=xi-yi normally distributed with expectation value . Hypothesis: H0: d=0 • Wilcoxon-test for paired samples - 2 paired samples (xi,yi) with expectation values 1 and 2 - Differences di=xi-yi symmetrically distributed  fulfilled if xi and yi have the same distribution shape. Hypothesis: H0: Page 20 1= September 2013 2 Siemens CT TIM CEE
  21. 21. Check of premises Before applying a hypothesis test, the differences (v0-v1 and c0-c1) have to be tested on normal distribution. Using the Kolmogoroff-Smirnoff test, we receive the following result: H0: Variable has a normal distribution. v1: click-through ratio new variable p-value v1-v0 0.04558 c1-c0 0.002431 v0: click-through ratio old c1: conversion rate new c0: conversion rate old =5%  no normal distribution in both cases (views, comments) Therefore we have to use a test which does not require normal distribution  Wilcoxon rank sum test. Page 21 September 2013 Siemens CT TIM CEE
  22. 22. Check of premises Symmetrical Distribution of differences v0-v1 and c0-c1: Page 22 September 2013 Siemens CT TIM CEE
  23. 23. Hypothesis Test: Principle of the Wilcoxon rank sum test Wilcoxon rank sum test (U-test for paired samples): Example for n=8 UR v0 v1 dv=v1-v0 rank for dv>0 rank for dv<0 1 0.02 0.02 0 - - 2 0.01 0 -0.01 3 0.01 0.10 0.09 7 4 0.06 0.13 0.07 6 5 0.03 0.04 0.01 1.5 6 0.11 0.15 0.04 5 7 0.06 0.08 0.02 3 8 0.03 0.06 0.03 4 1.5 R+ = 26.5 R=min(R+, R- )=1.5 Critical value for n=7 (UR1 excluded), =5%: Rcritical=2 R<Rcritical  H0: Page 23 September 2013 Siemens CT TIM CEE 1= 2 is rejected R- = 1.5
  24. 24. Test Results Results of Wilcoxon rank sum test: H0 p-value v0= v1 1 9.076e-07 c0= c1 0.4616 c0> c1 0.7718 c0< c1 Comments v0> v1 v0< v1 Views 1.815e-06 0.2308 Test result: Red: p<0.05  significant i.e. H0 is rejected. v0> v1 More views using old template. c0= c1 No significant change in number of comments. Possible explanations why there are more views of the old template: - Link to urgent request better visible. - Users used to old template. - Already enough information in e-mail  no need to view details. - Subjective impression of full information in new template. Page 24 September 2013 Siemens CT TIM CEE
  25. 25. Plots: response for old versus new template . Views in old (black) versus new (red) template Comments in old (black) versus new (red) template Page 25 September 2013 Siemens CT TIM CEE
  26. 26. Variable for comparison of old and new template Click-through ratio and conversion ratio: Problem of exclusion of zero values. Histogram: Conversion rate ratio convnew/convold 5 12 Count of Urgent Requests with corresponding ratio Count of Urgent Requests with corresponding ratio Histogram: Click-through rate ratio ctrnew/ctrold 10 8 6 4 2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 Click-through rate ratio 4 3 2 1 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 1,1 1,2 1,3 1,4 1,5 1,6 1,7 1,8 1,9 2 >2 Conversion rate ratio Page 26 September 2013 Siemens CT TIM CEE
  27. 27. Variable for comparison of old and new template Using differences v1-v0 and c1-c0 instead of quotients v1/v0 and c1/c0  zero values do not have to be excluded. Page 27 September 2013 Siemens CT TIM CEE
  28. 28. New First-Timers • Considering only subgroup receiving the new template: Is there a correlation between the number of “new first-timers” (NFT) and the number of (a) views (V1)? (b) comments (C1)? (a) H0: r(V1,NFT) = 0 (b) H0: r(C1,NFT) = 0 Kolmogoroff-Smirnoff test yields that number of new first-timers NFT is not normally distributed (p= 7.936e-10)  using Spearman„s or Kendall„s correlation coefficient. Page 28 September 2013 Siemens CT TIM CEE
  29. 29. New First-Timers • Considering only subgroup receiving the new template: Is there a correlation between the number of “new first-timers” (NFT) and the number of (a) views (V1)? (b) comments (C1)? Test results: case variables (a) V1,NFT (a) V1, NFT (b) C1,NFT (b) C1, NFT method r Spearman 0.3939 Kendall p-value 0.0017 0.2919 0.0015 Spearman 0.3976 0.0015 Kendall 0.3012 0.0019  H0 is rejected in every case (p<0.05).  significant positive correlation  Number of new first-timers related to number of views and comments: The more new first-timers, the more views and comments. Page 29 September 2013 Siemens CT TIM CEE
  30. 30. Sample Characteristics Improvement suggestion for novel split test Proposition of sample selection for next split test: • Existing TechnoWeb users are randomly split into two equally sized groups A and B. • Every new TechnoWeb user is assigned group A or group B randomly with a probability of 50% for each group. • Group A always receives the old, group B always receives the new template. • First time views don‟t have to be investigated separately by this approach, because they are more clearly distinguished from the beginning. Page 30 September 2013 Siemens CT TIM CEE
  31. 31. Results and Recommendations The following results were obtained: • No significant change in number of comments in new versus old template. • More views in old than in new template. • The more users receiving the new template for the first time, the more views and comments. • Statistically relevant number for comparison of old and new template: R=min(R+, R- ). Critical R varies according to sample size. Suggestions: • Use v1-v0 and c1-c0, respectively, instead of v1/v0 and c1/c0, in order not to exclude zero-answers. • Sample selection: randomly choose 50% that always receive old template, 50% that always receive new template and stick to that selection. Page 31 September 2013 Siemens CT TIM CEE
  32. 32. Agenda Page 32 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  33. 33. Learning: The new template has three usability problems • The overall performance of the new template is inferior to the old template • Identified cause: • An important eye-catcher, the assignment of a monetary value to the problem, is less visible in the new template, resulting in a decreased click-through rate. • Suspected causes: (to be validated in the next build-measure-learn cycle) • The prominently placed call-to-action in the new template might be less inviting for users – most do not want to comment right away. • The Link “Show Urgent Request” is much less visible in the new template • Part of the reduced click-through rate in the new template could be due to the content being presented in an easily-readable way. Old Template and Page 33 New Template and September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  34. 34. Moving key elements to the side decreased their effectiveness Page 34 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  35. 35. Conclusion Specific General • Split-testing a new feature is worth the time and effort. • Features that do not positively influence customer behavior should not be implemented. • The initial time investment in the first split is offset by knowledge gained on how to efficiently set up a split test. • Even though initially the problem looked simple, regular statistical text-book knowledge was not sufficient for the statistical significance analysis. • Initial negative results should not kill a project. Instead, iterative improvement will lead to a product that consumers will appreciate. • Consulting a professional statistician from the planning phase of the split would have saved much time and effort, and allowed to measure in a more focused way. Page 35 September 2013 Siemens CT TIM CEE Unrestricted © Siemens AG 2013. All rights reserved
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×