How to avoid being fooled by percent changes


Published on

A fictional tractor building company looks to understand why sales have decreased in the past year. Comparing the performance of two sales guys based on control charts yields very different results than a traditional "year-over-year" percent change analysis.

Published in: Business, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How to avoid being fooled by percent changes

  1. 1. What the board room can learn from the shop floor or, how to avoid being fooled by percent changes By Ben Jones
  2. 2. Meet Jim
  3. 3. (Jim says Hi)
  4. 4. Jim runs a company selling…tractors.
  5. 5. Tractor Sales aredown 3.6% this year -3.6%
  6. 6. Jim wantsanswers…
  7. 7. Meet Jim’s two sales guys, Joe and LarryJoe Larry
  8. 8. Larry’s numbers are up 2.9%… +2.9% Larry
  9. 9. Joe’s numbers are…not. -8.3%Joe
  10. 10. What should Jim do?
  11. 11. (Jim is polishing up his resume…)
  12. 12. but what if it was all just a big misunderstanding?
  13. 13. Instead of justlooking at % change “Year Over Year”… -3.6%
  14. 14. …what if we plottedunits sold by month?
  15. 15. and what if we applied some brand new techniques?Walter Shewhart written in 1939
  16. 16. to understand thevariation in the data: “Special Cause” variation UCL – Upper Control Limit “Common Cause” variation x-bar (or average) LCL – Lower Control Limit “Special Cause” variation
  17. 17. So wait, only LAST month is a “down” month…
  18. 18. …and WHO do you think caused THAT?Joe Larry
  19. 19. Probably Joe, right? -8.3%Joe
  20. 20. But if we look at Joe’snumbers over time… 2009: 96 Units -8.3%
  21. 21. But if we look at Joe’snumbers over time… 2010: 88 Units -8.3%
  22. 22. We see no statistically significant change! “Common Cause” variationJoe
  23. 23. But Larry can’t be thereason, his numbers are up, right? +2.9% Larry
  24. 24. Let’s look at Larry’snumbers over time… 2009: 69 Units
  25. 25. Let’s look at Larry’snumbers over time… 2010: 71 Units
  26. 26. Looks like Larry has some explaining to do about December… “Special Cause” variation Larry
  27. 27. These two approaches telltwo very different stories… Percent Change Control “YOY” Charts
  28. 28. Blaming Joe would bemistaking NOISE for a SIGNAL
  29. 29. Rewarding Larry would be missing the opportunity to identify a SIGNAL
  30. 30. Either way, Jim’s business loses an opportunity to LEARN and GROW…
  31. 31. …and the fate of oursales guys’ careers hangs in the balance… Joe Larry
  32. 32. The Morals of the Story:• DON’T just compare % change• DON’T assume a % change isstatistically significant just because youdon’t like it.• DO use a Control Chart to filter outnoise in a data set and identify signals• NEVER adjust the y-axis to cross thex-axis at anything other than 0 (did youcatch that trick? The column chartsgrossly over-exagerate the changes)
  33. 33. For bar charts, always set the y-axis to 0y-axis starts at 157 y-axis starts at 0 Appears to be 4X less
  34. 34. Got all that, Jim?
  35. 35. More about Control Charts (in business) • Understanding Variation – Donald J. WheelerPresentation by Ben Jones