Method, practice, and results. Presentation for UE Poznań, Department of Statistics, 26th November 2020

1. Diagnoza + - method, practise and results
Diagnoza + - method, practise and results
Magdalena Smyk-Szyma´nska and Lucas van der Velde
Warsaw School of Economics
FAME|GRAPE
UE Pozna´n
November 2020

2. Diagnoza + - method, practise and results
Introduction
Motivation
Two sources of inspiration:
Polish labor market and pandemic - lockdown consequences
(March-April)
main source of data: BAEL - next wave available in August
forced change in GUS method: CAPI transitioning to CATI
additional modules: life & work satisfation, norms, etc.
Online surveys in research
surge in online surveys
Wage Indicator Project: Smyk, Tyrowicz, van der Velde (2021) ”A
Cautionary Note on the Reliability of the Online Survey Data: The
Case of Wage Indicator”, Sociological Methods & Research

3. Diagnoza + - method, practise and results
Introduction
Diagnoza +
Joint initiative of several research institutions
→ University of Warsaw, FAME|GRAPE, CASE, CeNEA, IBS, EY
Goal: provide timely information on policy relevant variables
First survey draft → 27-30 April 2020
First survey sent → May 6th 2020
First report → May 31st 2020
All reports are available on our website diagnoza.plus

4. Diagnoza + - method, practise and results
Recruitment
How did participants find out about survey?
ANSWEO
“Volunteers”
Media appearances
Snowball sampling
Social media adds
Participants:
can join at any time → Filling entry survey (modelled after BAEL)
are contacted in specific months → Fill specific modules (PANEL)

5. Diagnoza + - method, practise and results
Recruitment
Rewards
Answeo → “normal” payment (0.1-0.3 USD) + every x person
receives a y price
“Volunteers” → every x person receives a y price
Generosity increased over time
From 50PLN for every 50th person to 15 PLN every 10th
Experimenting with rewards (when, how, how much)

6. Diagnoza + - method, practise and results
Recruitment
Attrition preventing
We encourage people to participate in several surveys.
Respondents volunteer by providing their contact
We send 3 (4) emails (1 invitation + 2(3) reminders)
total response rate ( for wave 4, only panel): ∼ 31%
response rate (1 invitation, 2 days): ∼ 18% (58% of final sample)
Participation:
Participated in the survey: approx. 30 000
Completed the survey: 13 258
Agreed to participate in a panel: 7 106 (some flawed)
Participated twice: 3 184
Participated three or more times: 4 965

7. Diagnoza + - method, practise and results
Recruitment
How do we fare by international standards
Examples of COVID-19 surveys:
Baert (Ghent) COVID IP Belot D+
(Edinburgh)
Where: Flanders UK, DE, CN, KR, JP, Poland
US IT, US, UK
Sample: ∼3,8k ∼4-5k per cn ∼1k per cn ∼5k per wave
How: online online online online
survey comp survey comp
Rep.: age, gender, age, gender, age, gender, age, gender,
educ region, educ, hh income region
[s] occ [s] [s] [w]
When: 3rd week April 9-14 4th week May - ...
of March of April
special COVID modules in panel studies such as SOEP, LISS and GIP

8. Diagnoza + - method, practise and results
Diagnoza + and BAEL
Is D+ different from BAEL?
Short answer
Yes
Long answer
Our research design is different.
→ Sample is not representative of Polish population
→ Sample frame: working age individuals who are neither retired nor
studying
→ Non-probability sample: little control on who joins the survey.

9. Diagnoza + - method, practise and results
Diagnoza + and BAEL
Comparing D+ to 2019 Q2 BAEL
Last year LMS matches well (almost perfectly) to 2019 Q2 BAEL
LMS
Minor differences in other characteristics:
More female - difference is shrinking
Younger →∼ 5 years
Higher proportion with tertiary studies
More likely to work on finance-related occupations
Concentrated on Mazowieckie - difference is srhinking

10. Diagnoza + - method, practise and results
Diagnoza + and BAEL
Obtaining weights to rebalance population
Goal weight our sample such that D+ looks like BAEL population
Two methods :
Covariate balancing propensity scores
Entropy balancing weights
Both methods successfully balance samples across all covariates

11. Diagnoza + - method, practise and results
Diagnoza + and BAEL
Covariate Balancing Propensity Scores (CBPS)
Introduced by Imai and Ratkovic (2013)
Main idea: use FOC from logit/probit models to derive weights
Estimator is derived from the treatment literature
Treatment → observation coming from BAEL
Reweight control units (D+) to balance covariates
Main idea: use FOC from logit/probit models to derive weights
BUT method is not fully dependent on propensity scores
Resilient to misspecifications

12. Diagnoza + - method, practise and results
Diagnoza + and BAEL
CBPS: stepwise derivation
Assume that
Pr(T = 1|Xi ) = πβ(Xi ) (T = 1 ⇒ BAEL = 1)
The FOC’s to maximize likelihood
1
N
N
i=1
Ti πβ(Xi )
πβ(Xi )
−
(1 − Ti )πβ(Xi )
1 − πβ(Xi )
= 0
we can replace πβ(Xi ) with any f (x) = ˜Xi and equality holds
˜Xi = Xi ⇒ balances means

13. Diagnoza + - method, practise and results
Diagnoza + and BAEL
CBPS: stepwise (II)
FOC’s in expectation:
E
Ti
˜Xi
πβ(Xi )
−
(1 − Ti ) ˜Xi
1 − πβ(Xi )
= 0
Sample analogue:
1
N
N
i=1
Ti − πβ(Xi )
πβ(Xi )(1 − πβ(Xi ))
˜Xi = 0
BUT these weights balance BOTH samples
→ modify equations to weight control units only

14. Diagnoza + - method, practise and results
Diagnoza + and BAEL
CBPS: stepwise (III)
FOC’s in expectation:
E Ti
˜Xi −
πβ(Xi )(1 − Ti ) ˜Xi
1 − πβ(Xi )
= 0
Sample analogue:
1
N
N
i=1
N
Nc
Ti − πβ(Xi )
(1 − πβ(Xi ))
˜Xi = 0
1
N
N
i=1
N
Nc
Ti − πβ(Xi )
(1 − πβ(Xi ))
These are our weights
˜Xi = 0

15. Diagnoza + - method, practise and results
Diagnoza + and BAEL
Empirical application
1 Trim BAEL sample
Exclude observations who are not in D+
Keep one randomly selected member from each household
2 Use CBPS to recover weights
Match on: gender, age, voivodeship, last year LMS, last year
industry, hh size, urban status
3 Compute statistics of interest on reweighted D+ sample
4 Bootstrap 1-3 to build CI

16. Diagnoza + - method, practise and results
Weights description
Balancing properties
BAEL D+
Before After
Female 0.5283 0.5659 0.5284
Household size
2 members 0.5070 0.2894 0.5069
3+ members 0.3331 0.5753 0.3331
Urban setting
100k-500k cities 0.2468 0.2765 0.2468
<100k cities 0.0537 0.0601 0.0538
rural areas 0.2237 0.1517 0.2239
Age†
42.1921 37.4333 42.1034
Note: D+ estimates correspond to the fourth report (forthcoming),
BAEL estimates correspond to Q3 2018 and Q3 2019. † a discrete
version of the variable was used for constructing weights

17. Diagnoza + - method, practise and results
Weights description
Histogram02.0e−044.0e−046.0e−048.0e−04.001
Density
0 1000 2000 3000 4000 5000
Estimated weights
Descriptive statistics
Mean 1000.00
SD 1295.45
p25 261.81
p50 613.33
p75 1257.29
p90 2329.10

18. Diagnoza + - method, practise and results
Weights description
Robustness checks - correlations
What happens if we ...?
Modify weighting variables
Exclude ly info 0.6879
Exclude ly indus 0.7654
Exclude age 0.6044
Exclude voivodeship 0.5645
Plus education 0.9076
Change estimation method
Estimate Ebalance 0.7729

19. Diagnoza + - method, practise and results
Results: how D+ enrichens the debate
Unemployment rate

20. Diagnoza + - method, practise and results
Results: how D+ enrichens the debate
Flows (August 2019 - August 2020)

21. Diagnoza + - method, practise and results
Results: how D+ enrichens the debate
Working from home

22. Diagnoza + - method, practise and results
Results: how D+ enrichens the debate
How did firms adjust to pandemic
0 .2 .4 .6 .8
odesetek pracowników og?oszaj?cych danej starania
(pytanie wielokrotnego wyboru)
restrukturyzacj? zada?
urlop / praca z domu
restrukturyzacja zespo?ów roboczych
umo?liwiaj?c zachowania odst?pu
dostarczenie masek
dostarczanie ?rodków dezynfekuj?cych
Jak pracodawcy reagowali na pandemi?

23. Diagnoza + - method, practise and results
Concluding thoughts
Plans for the future
Next report to be released this week
How COVID changed business practices & work environment
Next survey: most likely in January

24. Diagnoza + - method, practise and results
Concluding thoughts
Summary
D+ is a valuable (only) tool to study LM effects of pandemic in
Poland
Survey focuses on those more likely to be affected
CBPS effectively balances characteristics in D+ & BAEL
recent surveys: targeted recruitment to improve balance ex ante
Still descriptive measures = to BAEL → Why?
Sample selection in BAEL 2019 = 2020
Different definitions (unlikely to play a major role)
Differences can be explored when BAEL micro data becomes
available

25. Diagnoza + - method, practise and results
Concluding thoughts
Thank you and
we are happy to take questions!
w: diagnoza.plus, grape.org.pl
t: grape org
f: grape.org
e: msmyk@grape.org.pl, lvelde@grape.org.pl