Extract of a "what if...?" proposal for a tool for operations decision making in the Online Consumer Panels sector based on computational-intensive methods. A first demo exists at github.com/evaristoc

1. Computer-
Simulated Online
Access Panel Demo
Simulation tools for panel management:
Sampling Operations Analysis

2. ● Excerpt of experiment results using computer-based simulations
sketching sampling operations
● Exercise purposed to introduce / explore how and where simulation
tools could better serve information needs
● Experiments: sampling operations styles based on business case
parameters (eg used response rates for sample size calculation,
fieldwork period within a week and “incidence”) for a annual project
with a total target evenly spread in a weekly fashion
● Objective: to compare efficiencies of the sampling styles measured as
effects on potentially chargeable outcomes (eg number of undesired
types of responses different to completed surveys (QF), number of
weekly sample pulls to reach target, and total invites (LOADED) at
EOY)
Introduction

3. Demo: Sample Efficiency
How to decide the best
balance between
Panel Efficiency
(to avoid panel waste) vs
Labour Costs
(required sample pulls)?

4. Demo: Sample Efficiency
Hi Earth Team: This is the Project Manager X from Planet Ork! I am in trouble with the tracker XZ12W3 -
the project has been commissioned for another year but:
1. we have been pulling too many samples and we need to reduce them
2. we are using the universe response rate mean (21%) to calculate sample size
3. we have 4 days for each weekly fieldwork but there are frequent delays in the starting day
OK!!!
Don't Panic!
After looking some additional data, the team
decided to compare 4 different experiments
and see what happens...

5. Demo: Sample Efficiency
In the online consumer panel sector, past “response rate” or rr has typically been a parameter used to
calculate sample sizes; however the form and origin of the rr calculation could change to fit different needs
or constraints for example available data, project complexity or precision (eg. R. M. Groves, “Survey errors
and Survey Costs”, Wiley Series); the rr’s here named are related to the business case
The “incidence” is an estimate of the penetration of surveyed products or services in the population at a
given time.
The “eliminations” refers to delaying the respondent participation in a repeated survey during a period of
time in order to mitigate recall bias.
Although public, not all data or details about these experiments are publicly available. This author reserves
the right of disclosing information about this work.
Experiment #1 Experiment #2 Experiment #3 Experiment #4
Days in Fieldwork
(DFW)
fixed 4 days fixed 4 days variable -
min. 1 max. 4
variable -
min. 1 max. 4
Response rate (rr)
*
rr1 proposed rr2 rr1 proposed rr2
Incidence ** /
eliminations ***
pre-defined / yes implicit /
yes
pre-defined /
yes
implicit /
yes
Observed
result
lowest pulls
highest QFs
average pulls
average QFs
highest pulls
average QFs
average pulls
lowest QFs
*
**
***
Obs

6. Demo: Sample Efficiency
Exp1:
completes: 5200
quotafulls: 1274
loaded: 33826
Exp2:
completes: 5200
quotafulls: 883
loaded: 34917

7. Demo: Sample Efficiency
Exp3:
completes: 5200
quotafulls: 961
loaded: 48360
Exp4:
completes: 5200
quotafulls: 867
loaded: 46295

8. Project Manager X? This is the Earth Team. Our advice is as follows:
1. If you are certain of having 4 days of FW, and QFs are not an issue,
continue as you are doing.
2. If your FW varies (due to delays, etc.) then you MUST use differential
response rates.
Again, remember you can use R2D3 system to calculate the proper sample
sizes.
Let the Power be with you!
Thanks Earth Team!
(And the phrase says:
"May the FORCE be
with you!")
????....
Demo: Sample Efficiency

9. A Panel Simulator can:
1. Support business decisions by simulating
a. Panel behaviour and participation
b. Cost implications against expected outcomes
c. Operational requirements vs. client requirements
2. Add core competences in
a. Knowledge of online panel behavior against multiple
possible scenarios
b. Allowing to take advantage of capital margins
What does panel simulation do?