How to Avoid Wasting Time at Forecasting (Whitepaper)


Published on

We spend far too many organizational resources creating our forecasts, while almost
invariably failing to achieve the level of accuracy desired. The whole conversation needs
to be turned around. We should be focusing much less attention on modeling and
forecast accuracy and much more on process efficiency and effectiveness. We must
also consider alternative ways to answer the business questions that, out of habit,
we rely on forecasting alone to address. For more info:

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How to Avoid Wasting Time at Forecasting (Whitepaper)

  1. 1. How to Avoid Wasting Time at Forecasting CONCLUSIONS PAPERInsights from a webinar in the Applying Business Analytics Webinar SeriesFeaturing:Mike Gilliland, Forecasting Product Marketing Manager, SAS
  2. 2. SAS Conclusions PaperTable of ContentsThe Futility of Forecasting? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Trying to Predict a Random Future. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Forecast Value Added Analysis – A Lean Approach to Forecasting 2 . . A Simple Example of FVA Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3The Practical Value of FVA Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Eliminate Ineffective Steps in the Forecasting Process. . . . . . . . . . . . . 4 Fairly Compare Forecasting Performance. . . . . . . . . . . . . . . . . . . . . . . 5Getting Started with FVA Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Step 1: Map Your Overall Forecasting Process. . . . . . . . . . . . . . . . . . . 6 Step 2: Collect the Necessary Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Step 3: Analyze the Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Step 4: Report the Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Step 5: Interpret the Results and Take Action on the Findings. . . . . . . 8Industry Adoption of FVA Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 . .Closing Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10About the Author. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10About SAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11For More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
  3. 3. How to Avoid Wasting Time at ForecastingThe Futility of Forecasting? The job of the forecast analyst at Fair Coin Toss Inc. is to predict the frequency of heads vs. tails in 1,000 daily tosses of a coin. The forecaster’s predictions have been only 50 percent accurate so far. Management isn’t happy. With so much data and sophisticated software available, surely the forecasts could be 60 percent accurate or better. More data must be the key, management said. So they had the analyst augment his forecasting models with data about the physical attributes of the coin, the atmospheric and environmental conditions in which the coin was tossed, the performance attributes of the automated coin-tossing machine, the delay between coin tosses, and the rebound characteristics of the surface on which the coin was tossed. They organized a management committee to review the forecasts and tweak them based on their own experience and insights with coin tossing. Still the forecasts were only 50 percent accurate.The tossing of a fair coin to land heads-up or tails-up is a random occurrence, with a50-50 chance of one or the other. No amount of new data, past data, analyst overrideor management intervention can improve the odds of predicting whether the next tosswill land as heads or tails.So the first aphorism of forecasting ought to be that forecasting is a huge waste ofmanagement time – or rather, that it can waste a huge amount of management time.This is not to say that forecasting is pointless and irrelevant. It doesn’t mean thatforecasting isn’t useful or necessary to run our organizations – and it doesn’t mean thatmanagers shouldn’t care about their forecasting issues, nor seek ways to improve them.It simply means that the amount of time, money and human effort spent on forecastingis not commensurate with the amount of benefit achieved – that is, improvement inforecast accuracy.We spend far too many organizational resources creating our forecasts, while almostinvariably failing to achieve the level of accuracy desired. The whole conversation needsto be turned around. We should be focusing much less attention on modeling andforecast accuracy and much more on process efficiency and effectiveness. We mustalso consider alternative ways to answer the business questions that, out of habit,we rely on forecasting alone to address. 1
  4. 4. SAS Conclusions PaperTrying to Predict a Random FutureAlthough we live in an uncertain and largely unpredictable world, we prefer to operate The goal of our efforts shouldwith an illusion of control. No matter what kind of behavior or activity we are trying to be to develop forecasts asforecast – be it customer demand, financial costs and revenue, call center activity, loandefaults, insurance claims, or whatever – we think a bigger computer, a fancier model accurate as anyone canand a more elaborate process are all we need to get better forecasts. Unfortunately, reasonably expect them to be –the world doesn’t work that way. given the nature of what we areAs management at Fair Coin Toss Inc. eventually had to concede, forecast accuracy trying to forecast – and to dois largely determined by the nature of the behavior we are trying to forecast – its this as efficiently as possible.forecastability. If the behavior is smooth and stable, we should be able to forecastit accurately with simple methods. However, if the behavior is wild and erratic – orcompletely random, like heads vs. tails – there may be little hope of generating accurateforecasts, no matter how sophisticated our methods or how much effort we put into it.Therefore, the goal of our efforts should be to develop forecasts as accurate as anyonecan reasonably expect them to be – given the nature of what we are trying to forecast –and to do this as efficiently as possible.Forecast Value Added Analysis – A Lean Approach toForecastingLean is all about identifying and eliminating the wasted efforts in any process. A “FVA is the leanmethod called forecast value added or FVA analysis is a way to apply the lean approach manufacturing mind-setto forecasting. FVA is used to find those process activities that are justwasting time – that are failing to improve the forecast, or are even making it worse. applied to forecasting…” Tom WallaceForecast value added is defined as the change in a forecasting performance metric Author and supply chain thought leader– whatever metric you happen to use – that can be attributed to a particular step orparticipant in the forecasting process. Essentially, FVA is comparing the results of aprocess activity to the results you would have achieved without doing the activity.FVA can be positive, showing that you are adding value by making the forecast better.Or FVA can be negative, indicating that whatever you are doing is just making theforecast worse.FVA analysis is used to identify and mercilessly eliminate the non-value-adding activitiesin your forecasting process, so you can streamline the process and redirect the non-value-adding efforts into more productive activities. For instance, it might mean havingyour sales people out selling rather than trying to forecast future sales. As an addedbonus, when you eliminate the activities that are just making the forecast worse, youcan actually achieve better forecasts with less cost and effort.2
  5. 5. How to Avoid Wasting Time at ForecastingA Simple Example of FVA AnalysisTo understand FVA analysis, consider a very simple forecasting process, shown inFigure 1. Here, historical demand feeds into forecasting software, which generates whatwe call the statistical forecast. Then, an analyst reviews the statistical forecast and canmake a manual adjustment.FVA analysis compares the accuracy of the statistical forecast (generated by modelingsoftware) to the analyst’s override – and compares both forecasts to a “naive” forecastgenerated by very simple methods. Demand Statistical Model History Analyst OverrideFigure 1: The simple forecasting process.FVA analysis is the application of fundamental scientific method to the businessforecasting process. We start with a null hypothesis – that our forecasting process hasno effect on forecast accuracy. We then gather data to determine whether we can rejectthis null hypothesis.What we are doing is analogous to evaluating the safety and efficacy of a new drug ormedical treatment. For example, we find 100 people with colds and randomly dividethem into two groups, giving one group the new cold remedy, and the other group aplacebo. We then evaluate their recovery to see if those who had the new remedy getbetter faster.In FVA analysis, a naive forecast serves as the placebo. The naive forecast must besomething that is simple to compute, requiring the minimum of effort and manipulationto prepare a forecast. For example:• The random walk or “no change” model just uses your last known actual value as the future forecast. If you sold 12 units last week, your forecast for this week is 12 units. If you sell 15 units this week, your new forecast for next week becomes 15 units, and so on.• For the seasonal random walk, you use the same period from a year ago as the forecast for this year. Thus if you sold 35 units in October 2012, your forecast for October 2013 would be 35 units.• A moving average or other simple statistical formula is also suitable to use as your naive model – being within the spirit of simple to compute with a minimum of effort. 3
  6. 6. SAS Conclusions PaperWhen we conduct FVA analysis, we compare the forecasts generated at various stagesin our forecasting process to our placebo, the naive forecast. If the process is doingbetter than the naive forecast, we are adding value. If we find that our process is doingworse than a naive forecast, we are simply wasting time and resources.FVA results are commonly displayed in a stair-step report, as we see in Figure 2. Thereis a row corresponding to each sequential step in the process – here the naive forecast,the statistical and the override. The second column shows whatever metric we are usingto measure performance, typically MAPE (the mean absolute percent error), or accuracy.And the remaining columns show the pairwise comparisons. Process Step MAPE FVA vs. Naive Forecast FVA vs. Statistical Forecast Naive Forecast 25 percent Statistical Forecast 20 percent 5 percent Analyst Override Forecast 23 percent 2 percent -3 percentFigure 2: A basic FVA analysis asks, “Did the forecasting process do better thana naive forecast?”Here we see that the naive model achieved a MAPE of 25 percent. The statisticalforecast reduced the error by 5 percentage points, with a MAPE of 20 percent.However, while the analyst override had a MAPE 2 percentage points lower than thenaive model, it actually made the forecast worse by 3 percentage points compared tothe statistical forecast.In short, if you are doing better than a naive forecast, your process is adding value. If youare doing worse than a naive forecast, then you are simply wasting time and resources.It is not uncommon to find, as we see here, that human tampering with the process canmake the forecast worse.The Practical Value of FVA AnalysisEliminate Ineffective Steps in the Forecasting ProcessBy conducting a thorough and ongoing FVA analysis at your organization, you may beable to find process steps or participants that are failing to add value. The idea is tostreamline your process by eliminating those wasted efforts. Resources diverted awayfrom forecasting can then be redirected to more productive activities, like selling productor serving customers.• When FVA is negative – you can see that a process activity is making the forecast worse – then clearly that activity is unproductive and should be eliminated. FVA can also be used as an ongoing metric for tracking statistical model performance and indicating when models need to be recalibrated.4
  7. 7. How to Avoid Wasting Time at Forecasting By identifying and improving (or eliminating) non-value-adding activities, you can streamline your process and reduce the cost of resources invested in forecasting – Use FVA analysis to identify essentially getting better forecasts for free. non-value-adding activities: • Streamline the process by• When FVA is positive from one step to another, it can indicate that the process eliminating wasted efforts. step is adding value, as long as the incremental benefits justify the cost. • Direct resources to more productive activities.Fairly Compare Forecasting PerformanceAnother common use of FVA analysis is to compare performance between individual • Potentially achieve betterforecasters, product groups or organizations. Suppose you manage a group of three forecasts for free.analysts, and you will award a bonus to the best forecaster. Using traditional analysisrelying solely on MAPE or other traditional performance metrics, you would concludethat Analyst A, with a MAPE of 20 percent, deserves the bonus. But is this correct? Analyst MAPE A 20% B 30% C 40%Figure 3: Traditional analysis based on MAPE would say that Analyst A is thebest performer.Let’s look a bit deeper into the situation using FVA analysis. Suppose we find thatAnalyst A is responsible for forecasting sales for products that have long life cycles,no promotional activity and stable demand patterns. Sales of these products would berelatively easy to forecast. Even a naive model would have achieved a MAPE of just 10percent. FVA analysis shows that Analyst A is actually making the forecast worse by 10percentage points.Analyst B has products that are moderately difficult to forecast, with some seasonalityand promotional activity, a few new products entering the mix, and moderately volatiledemand patterns. Analyst B achieved the same MAPE as a naive model would haveachieved, so the forecast value added is zero.It turns out that only Analyst C added any value. Although C had the worst forecasterror at 40 percent, C had demand that was very difficult to forecast. A naive modelwould have only achieved a MAPE of 50 percent. So C had 10 percentage pointsof value added and deserves the bonus. 5
  8. 8. SAS Conclusions Paper Analyst Item Type Item Life Cycle Seasonality Promotions New Items Demand Volatility MAPE Naive MAPE FVA A Basic Long None None None Low 20% 10% -10% B Basic Long Some Few Few Medium 30% 30% 0% C Fashion Short Highly Many Many High 40% 50% 10%Figure 4: FVA analysis shows that MAPE alone can be misleading as an indicator ofanalyst performance.This example leads to a warning about one of the perils of benchmarking forecasting FVA analysis may reveal thatperformance. You cannot simply compare the MAPE or forecast accuracy achieved. having the lowest MAPE is notYou have to evaluate performance with respect to the underlying forecastability of the necessarily the same as beingdemand patterns. the best forecaster.MAPE is the most popular metric for evaluating forecasting performance, and itdoes tell you the magnitude of your forecast error. But MAPE doesn’t account forthe forecastability of what you’re trying to forecast. It doesn’t tell you the level ofaccuracy you should be able to achieve. And it doesn’t tell you anything abouthow efficient you are. In short, MAPE by itself is not a legitimate metric forcomparing forecasting performance.Getting Started with FVA AnalysisStep 1: Map Your Overall Forecasting ProcessThe process may be very simple, like the one on the left in Figure 5, perhaps with justa statistically generated forecast and a manual override – or it can be an elaborateconsensus process with lots of participation from various internal departments,customers and suppliers. Many organizations also have a final review step wheresenior management gets to change the numbers before approving them.6
  9. 9. How to Avoid Wasting Time at Forecasting Demand Causal Statistical Model History Factors Exec Sales Analyst Override Targets Mktg Collaboration/Consensus Customers Demand Statistical Model Finance Executive Review P&IC History Analyst Override Approved ForecastFigure 5: Begin your venture into FVA analysis by mapping process stepsand contributors.Step 2: Collect the Necessary DataIn a thorough FVA analysis, you capture and record the forecast every period, atevery step in the forecasting process: the naive forecast, your software’s statisticalforecast, other forecasts modified by manual overrides and consensus, or executive-approved forecasts.You want to gather this information at the most granular level of detail available, suchas by product and location. You also need to record the time bucket of the forecast,typically the week or month you are forecasting – and of course, the “actual” demandor behavior you were trying to forecast.Step 3: Analyze the ProcessHaving gathered the necessary data, now you can do FVA analysis – looking at howeach process step results in a positive or negative change in MAPE, weighted MAPEor whatever traditional metric you are using. It doesn’t matter which traditional metricyou use, including bias or forecast accuracy, because FVA analysis measures not theabsolute value of the metric but the degree of change at each step of the process.Comparisons may include:• Statistical versus naive forecast.• Analyst override versus statistical forecast. 7
  10. 10. SAS Conclusions Paper• Consensus versus analyst forecast.• Approved versus consensus.• Consensus participant inputs versus naive.Excel works fine for a quick one-time snapshot of FVA in a single period of time.However, ongoing analysis of a multistage forecasting process with a lot of productswill quickly grow into a large amount of data to store and maintain. You will want toautomate data collection and storage, so this is not something you do in Excel. Theentry-level SAS® Visual Data Discovery software easily handles huge FVA data sets,analysis and reporting, as well as dynamic visualization of FVA data.Step 4: Report the ResultsThere is no one fixed way to report FVA results, but a stair-step table is a good place tostart. On the left side you list the process steps or participants and their performance interms of MAPE or accuracy or whatever metric you are using. The columns to the rightshow the value added (or subtracted) from step to step in the process.For a more elaborate process, the report layout would be the same, except with morerows to show the additional process steps and more columns to show the additionalcomparisons between steps. You don’t have to report FVA for every possible pair offorecasts, but you should at least report every major pair in the chronological process.In addition to a stair-step report such as the simple example in Figure 2, graphicalpresentation of the data is best. For example, a histogram illustrating the distributionof FVA values for a group of products can be very insightful. Donald Wheelerpresents ideas for how to present data in his book about statistical process control,Understanding Variation: The Key to Managing Chaos.Step 5: Interpret the Results and Take Action on the FindingsBe aware that naive forecasts can be surprisingly difficult to beat. For example, a moving If you haven’t conducted FVAaverage is a perfectly appropriate forecasting model in some situations, and additional analysis and know that you arestatistical sophistication does not always generate a better forecast. beating a naive forecast, thenWhen a particular participant or step is not adding value, you should first try to maybe you aren’t.understand why. For example, do statistical models need to be updated so they willperform better? Do analysts need additional experience or training on when to makejudgment overrides and when to just leave the statistical forecast alone? Do certainparticipants in the consensus process bias results because of their own personalagendas? Do executives only approve forecasts that meet the operating plan andrevise those forecasts that are falling below plan?Be aware that as you conduct FVA for the first time, the results can be embarrassing toparticipants who are shown to add no value to the process. You may choose to shareresults privately and tactfully. Your purpose is to improve the process, not necessarilyto humiliate anyone.8
  11. 11. How to Avoid Wasting Time at ForecastingIt is also important to be cautious in interpreting your FVA results and not drawconclusions without sufficient evidence. You can’t just look at one period of data, ora short time frame, and determine whether you are adding value or not. Over short timeperiods, results may just be due to chance.Industry Adoption of FVA AnalysisFVA has been applied in companies across many industries, including consumerproducts, retail, pharmaceuticals, manufacturing, transportation, apparel, and food andbeverage. Major corporations such as Cisco, Intel, AstraZeneca, Newell Rubbermaidand others have gone public with their FVA results and the new ways they have appliedthe FVA concept. For example:• A premium home furnishings manufacturer used FVA to appeal to the competitive nature of its sales force. Sales reps were challenged to “beat the nerd in the corner” by improving upon the nerd’s statistical forecast.• When a large technology manufacturer looked at six years of historical data, FVA analysis revealed that half of forecasts failed to beat a naive model. The naive models were also less biased – neither chronically too high or too low.• At a major specialty retailer, analysts were making frequent adjustments to the forecast with each new bit of point-of-sales data from the stores. But FVA analysis showed that 75 percent of the analyst overrides failed to beat a moving average.• An automotive supplier found that, although management overrides did slightly improve forecast accuracy, the incremental gain might not be worth the cost of management time spent on the overrides.There is also academic research on the topic of human adjustments to forecasts.One study of four supply chain companies in the UK examined 60,000 forecasts,where 75 percent were manually adjusted. The study authors concluded that:• Small adjustments made almost no difference in forecast accuracy, which makes perfect sense. Small adjustments will not make the forecast much better or much worse, so they are probably not worth the effort.• Larger adjustments, particularly large downward adjustments that reduced the forecast, tended to add value by making the forecast more accurate.11 Source: “Good and Bad Judgment in Forecasting.” Fildes and Goodwin, Foresight, Fall 2007 9
  12. 12. SAS Conclusions PaperClosing ThoughtsForecast accuracy is limited by the nature of the behavior you are trying to forecast.While you cannot control the accuracy of your forecasts, you can control the processused and the resources invested. Overly elaborate forecasting processes with manymanagement touch points generally tend to make forecasts worse. More touch pointsbring more opportunities for people to add their biases and personal agendas – andcontaminate what should be an objective, dispassionate and scientific process.With FVA analysis, we can see whether forecast accuracy is being improved or erodedat each step in the process. We can improve or eliminate process steps that add novalue. If good software can give you reasonably accurate forecasts with little or nomanagement intervention, rely on the software and invest that management time inother areas that can bring more value to the company.About the AuthorMike Gilliland, Product Marketing Manager at SAS, has worked in consumer productsforecasting for more than 20 years in the food, electronics and apparel industries, aswell as a consultant. He wrote a quarterly column on Worst Practices in BusinessForecasting for Supply Chain Forecasting Digest and has published in Supply ChainManagement Review, Journal of Business Forecasting, Foresight: The InternationalJournal of Applied Forecasting, Analytics and APICS magazine. He is currently theForecasting Practice column editor for Foresight.Gilliland holds master’s degrees in philosophy and mathematical sciences from JohnsHopkins University. He deals with FVA analysis, worst practices and other forecastingtopics in his book, The Business Forecasting Deal: Exposing Myths, EliminatingBad Practices, Providing Practical Solutions. You can follow his blog, The BusinessForecasting Deal, at
  13. 13. How to Avoid Wasting Time at ForecastingAbout SASSAS Forecast Server is SAS’ flagship forecasting product, suitable for the forecastingneeds of even the largest enterprises. A high-performance forecasting engine provideslarge-scale, automatic forecasting from SAS code or via the SAS Forecast Studiointerface. SAS Forecast Server can diagnose the historical behavior of a time series,determine the appropriate class of models to deal with that behavior, and customizemodel parameters for each individual series. SAS Forecast Server has been adoptedat more than 500 organizations worldwide, across a wide range of industries.SAS is the leader in business analytics software and services, and the largestindependent vendor in the business intelligence market. Through innovative solutions,SAS helps customers at more than 60,000 sites improve performance and deliver valueby making better decisions faster. Since 1976 SAS has been giving customers aroundthe world THE POWER TO KNOW®.For More InformationTo view the on-demand recording of this information about events in the Applying Business Analytics Webinar view the Forecasting 101 on-demand access Mike Gilliland’s blog, The Business Forecasting view the on-demand webcast, Forecast Value Added Analysis: Step by download the SAS white paper, Forecast Value Added Analysis: Step by Business Forecasting Deal: Exposing Myths, Eliminating Bad Practices,Providing Practical Solutions, by Mike Gilliland is available through the SAS bookstore, and other booksellers.Follow us on twitter: @sasanalyticsLike us on Facebook: SAS Analytics 11
  14. 14. About SASSAS is the leader in business analytics software and services, and the largest independent vendor in the business intelligence market.Through innovative solutions, SAS helps customers at more than 60,000 sites improve performance and deliver value by making betterdecisions faster. Since 1976 SAS has been giving customers around the world THE POWER TO KNOW ® For more information on .SAS® Business Analytics software and services, visit SAS Institute Inc. World Headquarters   +1 919 677 8000 To contact your local SAS office, please visit: SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are trademarks of their respective companies. Copyright © 2013, SAS Institute Inc. All rights reserved. 106146_S95306_0113