In this project a model of an online consumer panel is proposed. The model was based on observations and empirical analysis. Then an agent-based artificial panel was developed based on that model. For demostration purposes, the panel was subjected to stress and other modifications to analyse the possible behaviour of an actual panel under similar circunstances. The presentation concludes with a short analysis of benefits and cost of this approach for online panel companies.

1. Alex’s life in SIMU LTD
Using simulations for
online consumer panels
operations analysis
advances of a model proposal

2. Project
● Previous work
■ slideshare.net/evaristoc
■ nl.linkedin.com/pub/evaristo-caraballo
● Objectives
○ to introduce concepts; sketch the model
■ Advances and modifications on version 2.0
○ to show potential and scope of a simulation
tool using a fictitious case

3. What is the company’s model?
Welcome to
SIMU LTD:
the Online
Panel!!

4. SIMU LTD asks people to subscribe at SIMU
LTD under company’s costs...
If they subscribe they will
receive emails inviting to
respond online
surveys in a portal...
… they could answer or not...
SIMU Ltd.
SURVEY Co .

5. SIMU LTD works hard in
keeping enough people “in
residence” willing to answer
the surveys of the client,
SURVEY Co., as well as
improving gross margin and
profits.
The set of subscribers
responding surveys is known
as THE PANEL.
SIMU LTD always strives to
understand how subscribers
turn into a better panel.

6. And this is Alex!
Alex has voluntarily joined SIMU
LTD. to answer surveys online.
Alex receives a “reward” for the
efforts.
As subscriber at SIMU LTD Alex is
usually referred as PANELLIST.
Alex shares certain gender and
age with other panellists
(“neighbours”) and has a demo-letter
(“a”, “b”, “c” or “d”) randomly
assigned.

7. Alex’s behavioural model (sketched)
Loyalty (desire to
stay)
- Comparison with other
panels
- Quality of service and other
perceived rewards
- Satisfaction; Joy; “What is
there for me?”
- Tenure (proxy)
Response (desire to initiate
and continue)
- Size and type of reward per survey
- Questionnaire length
- Quality of present and previous
questionnaire experience
- Topic, Frequency
Effort (“maximum per
week”; “yield”)
- `Planned´dedication time and
budget; Necessity, Availability
- Channel, Presentation (Exposure;
Accessibility)
- Season or other external events
All proposed variables
could interact to define
the response rate model
A simple probabilistic model based mainly on tenure and
effort was considered for this project

8. Alex’s response while active
u-Strategy m-Strategy
The more the expectations are
matched, the higher response
rate, tenure and max effort...
Unmatched expectations
Low effort-rr-tenure
(150 min/wk = 15 surv max)
● Alex is rational, knows what to expect and has already a
Matched expectations
High effort-rr-tenure
(300 min/wk = 30 surv max)
“strategy”
○ response is a function linked to a pre-assigned active tenure
● Assumptions and Constraints:
○ Alex is sensitive to number of surveys in portal up to a maximum
○ Alex is immune to number/quality of invites
○ the longer in the panel, the lower the initial* response
(*) Reflects empirical data

9. Nothing last forever and Alex will leave the
panel at some point, but when?
This is the period where Alex answers
some invitations
During inactive period Alex stops completely.
SIMU LTD automated system doesn’t know
exactly when this happens: it just “purges”
Alex after some consecutive weeks of non-responses
First Invite Last Response Purged
... ...
Active Tenure
(including seasonal
inaction)
(unsubscribed)
Inactive Tenure
(non-response no matter
number of invites)
Total Tenure
X X
NOTES: panellists reach a “Inactive Tenure” period
where they are most likely ignoring any invitation to
participate in the panel

10. Each member of any cohort* has specific
tenure, so panellists leave the panel at
different time
TIME wk
01
We begin
with a
relatively
large group
of recruits...
wk
02
wk
n-4
wk
n-3
wk
n-2
wk
n-1
wk
n
… many of
them leave
very soon…
others stay
longer...
… there are
these few
panelists
that last
longer ...
… reaching
a lower
churn rate
...
...
...
Alex Alex Alex
Alex Alex Alex
Alex
(*) Cohort: panellist that started same day

11. The mix of panellists who are active
or inactive at a given moment and
who could potentially receive
invitations to participate are all
together named...
THE (available) UNIVERSE

12. Invitation Model*
(*) For a sketch refer to a summary of version 2.0 at slideshare.net/evaristoc
1- In our model all panellists in the universe queue in a
random order to “respond” an also randomised list of
surveys for that “week”...
2- By turns, each panellist will be
exposed only to those surveys that
are still uncomplete, for which they
are selected or not excluded; we
say that they have a “portal”; they
can open the portal or not...
3- The process ends when all available
weekly surveys are completed or when the
instances in the universe are exhausted (i.e.
each panellist was presented once with the
respective portal)

13. What for questionnaires does
Alex find? SURVEY Co. projects
Ad-Hocs:
❏ One-time project that
last 1 week
❏ 70 completes
❏ Incidence: 20%
❏ Selection by demo
letter
❏ No Exclusion
Trackers:
❏ Same weekly project
for 1-3 years
❏ 40 completes
❏ Incidence: 10%
❏ Selection (fixed along
the project duration)
❏ Exclusion by previous
response status
Both last “10 min” and SIMU LTD gives 0,10 money-equivalent
per complete to the panellists

14. Now let’s talk about BUSINESS...
Issues being explored with the exercise
1. How does this particular model help to
determine a panel capacity* that could
affect business performance?
(*) Eg. max number of surveys that the panel would answer at a given time
2. Insights about business implications
when introducing changes that could
modify, e.g., panel response behaviour?

15. SIMU LTD. has been operating for
5 years to date delivering Survey
Co.’s surveys
Started with ad-hocs, adding
trackers from year 2
See weekly KPI developed for this
exercise in Annex A
Let’s see how SIMU
LTD has been doing...
OBS: SIMU LTD can correctly estimate
performance and invoice accordingly!

16. SIMU LTD. performance (5 years)
During the 5 years of operation
and with a fixed recruitment
effort SIMU LTD universe
trends to stabilise after the first
year.
OBS: in this project seasonality
was not evaluated
After stabilising, SIMU LTD
universe showed a weekly low
response rate (average num.
touches/num.surveys) for an
average of 11 surveys/week
per panellist’s portal.

17. SIMU LTD. performance (5 years)
With a stable revenue/cost ratio,
SIMU LTD keeps close to the
ideal gross margins*.
Higher portion of costs (75%)
due to survey costs.
Although only average 7% of
total costs, recruitment costs
threat earnings by growing at
10% per year!
(*) gross margins if not need of
buying PP services
Excluding first year, only 2% of
assigned completes were re-sold
to PP (average 2500
completes/year).

18. Increasing Gross Margins: How?
SIMU LTD has ideas for the next 2 years:
● Sales: doubling / tripling number of
weekly surveys after base price cuts
(see Annex B)
● Panellists: a campaign to extend the
permanence of u-strategists
(see Annex C)
Here the results of the analysis...

19. Sales increase by price cuts
Although gross margins
doubled, SIMU LTD wouldn’t
match the ideal gross margins
(only 60% during 7y).
Going from an average 75% to a
85% of total costs, survey costs
had a large impact...
… but now non-covered
completes become a source of
losses:
For every 30 additional
surveys/wk (every added 100k
completes/year) the fail-to-complete
quintuplicates.

20. And how did the respondents?
Simu ltd. typified respondents according to their
(active) tenure in bad, mid and good, took a
weekly sample of just-inactivated panelists at the 3
periods (1-5y, 6y and 7y). Then analysed the
performance of each respondent type per period.
Simu ltd. found that the
completion rate decreased,
no matter the respondent
type. The most affected was
the response of the good type.
The response rate also decreased, BUT actual
contribution of bad respondents lowered
substantially at each following period because their
lowered response rate AND their lower bound to accept
surveys per week, that forced a maximum.
Better respondents took the effort of weekly
checking the “leftover” of surveys other panelists “didn’
t have time to see”. Better respondent saw even more
surveys when the number of surveys tripled.

21. Sales + Targeting u-strategists
- The campaign affecting u-strategists will always increase median response rate and universe
size
- However the campaign has an insignificant and even negative effect over gross margins
whether combined with any kind of sales increase or not
- If no other changes are possible, doubling/tripling without campaign appears the best risk
to take when compared to the other proposals

22. What are SIMU LTD take-aways?
● Negotiating sample engagement (eg tenure,
“professionals”) composition based on
clients demands?
■ type of responsiveness and engagement is a
sample quality attribute for some researches!
● How OPERATIONS (eg. increase in surveys
per week) could affect the responsiveness?
● When thinking in a loyalty campaign, which
panellist(s) type should be targeted? why?
how? What is the best panellist model?
■ this point could be an STRATEGIC one…

23. Final Remarks
● Combines decision-making arguments
into single scenario visualizations
● When?
○ testing our assumptions; “what if…?”
○ taking existing tools a step further
○ prone to automation => good for reporting
○ explaining, even predicting
■ R&D... Training...
○ as support tool if actions/analyses are
costly, risky, complex
■ eg. “augmenting” vars./data; advanced stat.
○ data analysis, reliability, validity a pré

24. ANNEXES

25. ANNEX A.- Fictitious Weekly
Financial KPI’s for SIMU LTD
(period 2-5ys).

27. ANNEX B.- Experiment 1: Sales
increase by cutting prices

28. ● Cut is always over Base Price per complete at end
of previous year
● First cut (year 6) is for 90%; second cut (year 7) is
for 75%
● Doubling/Tripling means:
○ Doubling/Tripling number of fixed number of trackers at year
5 (7) as well as average ad-hocs at year 5 (24)
● The cut and the increase in sales take place
immediately at the beginning of each year
● Constraints
○ Only price is affected; the rest keeps unchanged
○ PP are also competitors; they offer similar prices per
complete as SIMU LTD
○ PP won’t match the price cut, keeping the same base price as
at year 5 during years 6 and 7

29. ANNEX C.- Experiment 2:
targeting u-strategists

30. ● Based on fictitious few weeks A/B experiment
under average last 4y conditions
● Targets are instances with 1-2 wks active tenure
● The campaign:
○ initial investment (10k) to be paid at the beginning of year 6
○ “reward” (6,- units) for new recruits who respond at least
once per wk, 3wks in a row, given at the end of week 3
● After starting experiment SIMU LTD expects:
○ Increase of 1-3.0% more respondents at second week
○ Increase of 3-5.0% at the third
● Constraints
○ Because SIMU LTD doesn’t know who the u-strategists are the
campaign is generic
○ Those instances already programmed for a tenure longer than
2wks are believed to be immune
○ It is assumed that the campaign doesn’t affect response rate:
only tenure