This presentation elaborates on the topic how people react to contextual information when forecasting in the presence of promotions. This talk highlighted the need for broader research on Forecasting Support Systems (FSS) and its combination with human interventions.

1. Experiments in Operations Management:
Information use in supply chain forecasting
Robert Fildes: L:ancaster Centre for Forecasting
With thanks to Paul Goodwin: U. Bath
Management School

2. Agenda
• Forecasting processes in practice: the S&OP process
– Using information in the S&OP process
• Researching the S&OP process
– The role of behavioural experiments
• Hypotheses on information use
• Experimental methodology
• Results and discussion
• Implications for research and practice

3. The S&OP process
Purpose:
• integrating the strategic, tactical and operations in
matching supply and demand
• Tuomikangas, N., & Kaipia, R. (2014). A coordination framework for sales and operations
planning (S&OP): Synthesis from the literature. International Journal of Production
Economics, 154, 243-262
• Very limited research: practitioner war stories
Focus here on the operational
• Information used in producing the sales forecast (a
reconciliation of demand and supply)
– Marketing, sales, logistics and finance
• Internal data much more commonly used than external
– Even in firms practising CPFR (Weller, M., & Crone, S. F. (2012). Supply Chain
Forecasting: Best Practices & Benchmarking Study. Internal Report. Lancaster Centre for
Forecasting.)

4. Sales and Operations
Planning
- Forecasting Process
• Statistical forecast
• Information from sales, market research, planning,
finance and logistics
• Incorporated into a final forecast from the
forecasters
• Delivered back to interested parties
• Judgment a key component

5. Data
Base
Statistical
Forecast
Functional
Forecast
Judgmental
Forecast
Demand Planner
Forecasting System
● Model A
○ Model B
Marketing
Sales
Logistics/ Production
Finance
Planning
The process (Sales and Operations Planning)
• Statistical forecast
• Information from sales, market research, planning and logistics, finance
• Incorporated into a final forecast from the forecasters back to interested parties
• Judgment is a key component for integration
Forecasting Process

6. A stylised view
Select &
run
forecasting
model
Logistics
reconciliation:
supply shortages
Combine
forecasts
using MI and
FF
Data
+ revisions
Market
intelligence (MI)
Functional
forecasts
(FF): Advice
Two forms of
information
• MI
• Advice

7. The revelation:
Judgment plays a key role in most forecasting processes
• It is unanalysed
Survey: The Forecasting Process
How are forecasts typically produced?
Fildes &
Goodwin 2007
Lancaster
2013 survey
i) Judgment alone 25% 15.6%
ii) Statistical methods exclusively 25% 28.7%
iii) An average of a statistical forecast and
management judgmental forecast(s) 17% 18.5%
iv) A statistical forecast judgmentally adjusted by the
company forecaster(s) 34% 37.1%

8. Reasons for using judgment
(Survey research)
8
Reason for using judgment
%
indicating
important
or
extremely
important
Promotional & advertising activity 51.4
Price change 45.5
Holidays 36.0
Insufficient inventories 33.3
Changes in regulations 32.7
Government policy 27.9
Substitute products produced by your company 22.5
Activity by competitors (promotions, advertising etc) 20.7
International crises 18.0
Weather 16.7
Insufficient inventories of competitors 16.2
Sporting events 11.8
Strikes 9.9
Other 36.4

9. A core problem in the supply chain
Promotional demand
- a ‘solved’ problem in marketing science but not in
reality
0
20
40
60
80
100
120
140
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
S
a
l
e
s
Period
Promotional sales and baseline
Total Sales
Baseline
• Time varying effects
• Uniqueness of events
• Incorporation of ‘soft’ information on
event effects
• Structural breaks due to new products

10. Why not rely on a statistical model?
Is a complete model possible?
• The Problems
– Too complex
– Incomplete data on many
drivers
– ‘Unique’ events
– No available statistical
expertise
– Management understanding
& acceptance
– Belief that managerial
expertise is best
Marketing factors
The Practical Solution:
Capture the unusual complexity by managerial judgement
( , ; )it it t it iSales f X e 

11. Evaluating the S&OP Process
Short-term KPIs
• ‘Profit’
• Inventory-service trade offs
• Sales forecasting accuracy
– Service!
• Functional biases:
Organisationally consistent
forecasts
– Finance, distribution,
operations
• Supplier relationships
Auditing organisational capabilities
(Moon et al, IJF, 2003)
• Functional integration
– in the organisation
– Across the supply chain
• Systems
– data base
– software
– support
• Approach and methods
– Techniques
– KPI measures (accuracy)

12. Behavioural research in Ops.
Management
Operations context and methodology in behavioural operations
(Croson et al., J. Ops. Management, 2013)
Forecasting
4% Supply Chain
27%
Experiment
32%

13. Researching the S&OP Process
- the need for triangulation
Issue
• Understanding organizational
processes
• Identification of biases,
inefficiencies
– In the results
– In the processes
• Information and systems use
• Improving effectiveness
Methodology
• Case studies
• Inter-organisational survey
• Case and statistical survey
– Statistical intraorg survey
– Case process analysis
– Behavioural experiments
• Inter-organisational survey
• Statistical analysis
• Behavioural experiments
• Action Research
• Behavioural experiments
• Normative models

14. Key research issues in understanding the
S&OP process
• How do forecasters use Market Intelligence and
‘Advice’ to produce the final forecasts ? Biases and
inefficiencies
– Case research (Goodwin et al.,2007)
– Statistical cross-org analysis (Fildes et al, 2009; Franses and Legerstree, 2009)
• Does the format in which information is presented
affect accuracy
A set of realistic grounded experiments to understand:
• The effects of different types of information and advice
• Any motivational effects from differential rewards
Research:
Practically important to improve the effectiveness of S&OP
Theoretically important how information is used in complex tasks

15. Methodological ‘triangulation’
Population of divers
supply chain companies
Processes, information use
and demand forecasting
Models of
information
use in
decisions
Questionnaire/
Stat explanatory
models
Hypotheses &
Normative
implications
Experiments
Case
observation/
action research
Experiments generate hypotheses to be further studies through survey and case
• Also structure descriptive (normative?) models and improved processes

16. Benefits/ disadvantages of experimental
research
• Control of experimental features
– Design and Use of information system
– Information available to affect outcome
– Comparative within subject (forecaster) use of information
• Disadvantages
– Subjects are not in a professional forecaster setting
– FSS is not commercial
But
• Experimental FSS designed with features commercially
available
• Students all have at least as much training as most
forecasters
• Motivation?

17. Process improvements:
Understanding information use.
Types of Information Available for Judgmental Adjustments in
Experiment
Time series
history
Statistical
forecast
Verbal / contextual
information
Positive Negative
Judgmental
adjustment
Reliability of past
stats forecast
Base rate
information
Personal &
Prior view

18. The experimental task: the null case
- paper under review in IJF
• Participants assumed role of a forecaster for a large company
which supplies a wide range of products to supermarkets.
• For a series of products – invited to adjust statistical sales
baseline forecasts for effects of forthcoming promotion.
• Invited to indicate reason for adjustment (from those
provided)

19. Information provided
Participants were provided with:
 a graph of sales & statistical forecasts for last 24 periods (this
included 1 previous promotion period)
 details of the product
 an indication that the mean promotion sales uplift (base rate)
was 50%
 between 0 and 4 positive & negative reasons why the promoted
sales might exceed or fall below the base rate
 Information delivered electronically. Email typical in companies (Weller,
Crone survey)

20. Details of reasons
Categories (with expert ratings of importance)
• Promotional spend (1=)
– These "3 for the price of 2" offers have not worked in the past for this type
of product. People just don't want to buy in these quantities.
• Campaign effectiveness (1=)
– Sales staff have been in discussions with the supermarkets. The good news
is that they have agreed to display the product prominently during the
campaign.
• Market research (3)
– Market research results relating to the promotion campaign have not
been encouraging according to the Marketing Manager.
• Weather (4)
– Weather conditions in the Midlands where this product is popular, should
help to boost sales substantially.

21. Process improvements:
Experiment
Aim:
• To understand how the interactions between supply chain actors
affect reliability and efficiencies
• The develop FSS support to test improvement strategies
Methods:
• Experiment with different forms of FSS and types of information
• Analyse driver interactions
Conclusions (so far):
• Interactions biased and highly inefficient
• FSS & S&OP likely to hinder
• Systems offering guidance can help

22. The experimental task

23. Example hypotheses
H1: Pattern matching -Larger sales achieved in the previous promotion will be
associated larger upwards adjustments.
H2: The closer the timing of the previous promotion to the current period the
larger the adjustment will be (salience).
H3: Series characteristics – e.g. Levels of noise in a series will not be
associated with the size of upwards adjustments.
H4: Forecasters will adopt a non-compensatory strategy when considering the
estimated effects of multiple items of contextual information.
• Alternatively, a compensatory strategy with negative information
counteracting positive
H5: Negative information will have a greater influence on forecast
adjustments than positive information.
H6: Statistical base-rate information on the average uplift will tend to be
neglected.

24. Simplified hypotheses – in response to reviews
H1: Adjustments to statistical baseline forecasts to take
into account forthcoming promotion effects will deviate
from base rates when information with no, or
unknown, diagnosticity is provided.
H2: Adjustments made to statistical baseline forecasts
to take into account the effects of forthcoming
promotions will tend to underestimate these effects
when information with no, or unknown, diagnosticity is
provided.

25. The Data Generating Process
1 10.2* 0.8* ( )
(80,120)
( (1,24)
25
)
t t t t t
t t t
t
Sales BaseLineForecast PromInd * Promotional Effect
BaseLineForecast Sales BaselineFor eecast P rturbation fo
Promotional Effect Uniform
PromPeriod Int Uniform
Pr
t
om
r

 
  
   
, 0
(0, ).
t t
t
Ind = 1 when t = PromPeriod otherwise
Normal stddev

The Simplifications:
• One past promotion per SKU in data base
• Promotions last one period
• Only one type of promotion
• No trend (but we tested for this)
• No seasonality (important in practice)
• Baseline forecasting model Research question:
• What are the company processes for
forecasting promotional uplift?
The Extensions?:
• History of past promotional forecasts
• Promotions dynamic
• Many types of promotion
• Seasonality (important in practice)
• Statistical forecasting model

26. Implementation of experiment
• Participants first asked what they thought typical % sales uplift would be
for a fast-moving consumer good that was being promoted (Prior
estimate)
• Trial run with 2 SKUs & feedback on accuracy plus assessment of why
previous promotion uplift had exceeded/fallen below 50%
• In main experiment they made forecasts for 12 SKUs in random order
• Reasons provided varied randomly between 0 and 4 with equal chances
of positive or negative information being provided
• Statistical baseline forecast randomly perturbed (to avoid collinearity)
• No feedback in main part of experiment
• Post-experiment questionnaire

27. Participants
112 Business & Management students at Bath, Bilkent & Lancaster
universities (all had studied forecasting).
Motivation
To control for motivational factors -randomly assigned to
A: rewarded when a promotion uplift exceeded 50%
B: rewarded for the accuracy of their forecasts
C: rewarded merely for participating in the experiment.
This led to a 3 (motivation type) between subjects x 12 (SKUs)
within subjects design.

28. Results
Prior expectations of uplift and uplift during experiment
Prior estimate
Adjustment during
experiment
Change in
estimate
Mean 50.8% 30.8% -20.0%
Median 50.0% 30.0% -15.0%
Supports H9 that base rate
will be discounted

29. Analysis based on linear mixed-effects
model
• Dependent: log(100+Adjust_Percent).
• Independent variables assumed to be random effects:
-Log (respondent’s prior estimate of promotional effects),
-Log (last forecast error),
-Log (uplift achieved in the last promotion)
-Log (baseline forecast for the promoted period)
-Timing of previous promotion.
• Independent variables treated as fixed effects class variables:
-Noise variance
-No. of positive and negative reasons (Pos-Neg)
• Points of high leverage removed

30. Effect Estimate p-value
Intercept 0.505 <.0001
ln(last uplift) 0.275 <.0001
ln (last actual) 0.037 0.0015
ln (last stats forecast error) 0.040 0.1051
ln(current stats forecast) -0.128 <.0001
ln(Prior) 0.035 0.0141
Noise Variance 0.021 0.0179
Timing 0.001 0.0357
Reasncat = -4 -0.107 <.0001
Reasncat =-3 -0.137 <.0001
Reasncat = -2 -0.114 <.0001
Reasncat = -1 -0.077 <.0001
Reasncat = 0 -0.078 <.0001
Reasncat = 1 -0.052 0.0014
Reasncat = 2 -0.023 0.1120
Reasncat = 3 -0.027 0.1004
H1 ✓
H3✓
✓
H2 ✓
H4 X
H3 X
H4 X

31. Participants appeared to adopt a
compensatory strategy

32. Compared to the normative model
No influence:
• Past sales
• Market Information
• Previous promotion and its timing
Forecast should be based on:
• Base line forecast
• Average past uplift (50%)

33. Post-Experiment questionnaire
Question Std.Dev.
Rating of overall knowledge of demand forecasting 2.77 0.86
Expectations of statistical forecast performance 3.03 0.77
The provided reasons had a direct influence on my
forecasts 3.46 1.07
Confidence in my final adjusted forecast 2.66 0.94
Motivation to engage with the task 3.40 0.98
Table 1 Questionnaire responses
Scale: (1) None / low expectations, to (5) High / high expectations -
depending on question

34. Overall
• Participants were aiming to match the previous promotion’s
sales
• This strategy was moderated by…
 recency of the previous promotion,
 level of sales achieved in the most recent period,
 level of noise in the series,
 difference between the no. of positive & negative
 reasons.
• The advertised base rate of 50% appeared to have been
discounted.
– H1 and H2 (new) supported)
– Moderating factors important (in contrast to normative model)

35. The Second Experiment – removing moderating
factors
• Participants were 30 executives
• Two treatments
– Previous promotion
– Reasons (2 positive, 0, 2 negative)
Aim:
• Confirmation/ Testing results of first experiment
• A check on ‘student participants’ vs experienced
executives

36. Results of Experiment 2
Treatment 1:
Previous
promotion
Treatment 2:
Reasons
Effect Estimate
(n=159)
p-
value
Estimate
(n=161)
p-
value
Intercept -4.830 <0.001 -2.946 0.001
ln(last promotion uplift) 0.678 <0.001 n/a n/a
ln(current stats forecast) 0.889 0.001 0.598 0.003
ln(Prior) 0.418 0.153 1.160 0.133
Low Noise 0.190 0.001 -0.022 0.608
No. of pos. minus no. of
neg. reasons n/a n/a 0.032 0.0416
Mean adjustment 51.9 46.6
Median adjustment 42.9 33.8
Model of promotional adjustment for the two treatments: ‘Previous
promotion’ and ‘Reasons’
All tests are one-sided except for those for Low Noise and the intercept.
Note that zero diagnosticity reasons lowered the adjustment
But the adjustment was higher than in experiment 1. Why – less distraction?

37. Why was 50% uplift consistently
underestimated?
• Most plausible explanation:
-The previous promotion effect and the statistical forecast both
acted as anchors.
- Estimated uplift tended to be set at a point between them.
Adjusted fcst
Stats baseline fcast

38. Indicating a main reason for adjustment
• Larger average upwards adjustment made by those who
chose to indicate which reason had the greatest influence on
their adjustment
• Median adjustments were 30% compared to 25%
• Engagement with Adjustment process important?

39. Conclusions
• Surprisingly wide range of information used by
participants
• Compensatory use of verbal information
– Information accumulates non-linearly
• Forecasters are inefficient in using the information
available to them
– Mis-weight historical information
– Mis-weight cross-functional information (MI)
– Mis-weight advice (statistical forecasts)
– Discount relevant prior information in favour of one
observation (sales in most recent promotion)

40. What does the experimental approach tell
us about the S&OP process?
• Nothing directly
But
• As a partial triangulation
– Case analyses show the integrative capabilities of a designed
S&OP process
– Statistical analysis of company case data shows biases and mis-
interpretations of promotional data and time series history
– Limited case data shows models and forecasts being adjusted to
incorporate information with biased results
– Now experimental evidence shows similar misunderstanding.
– Theoretical models have offered little!
The evidence now strongly supports the view that the S&OP
process will generally lead to mis-weighting of information
 What’s to be done?

41. Implications for research and practice
• Behavioural experiments productive
– But simulating reality remains challenging
• Motivation not captured
– More complex demand processes needed (in preparation)
• Richer evidence on the use of model based forecasts
– FSS including information modelled more fully (new experiment)
• AS considering experiments where the information is diagnostic
– More case-based research
• Dominant information/ actors
• In practice:
– Major requirement for feedback in commercial FSS
• Automatic identification of biases and ‘value added
– ‘Notes’ systems need to be included and designed to present and
summarise information

42. Issues:
• Increasing the diagnostic information available
• Different formats of information delivery important
– Quantitative vs enhanced qualititative vs combination
• Design FSS to respond
And
• What are the key new experiments
Comments