Presenter: Joe Reisinger, CTO & Co-Founder, Premise Data

1. @josephreisinger
@premisedata

3. WHAT PREMISE MEASURES

4. Bringing visibility to the world’s hardest-to-see places. 130 cities, 30 countries.

5. Modernizing
Economic
Measurement

6. “I have been constantly surprised at how little quantitative
information can be brought to bear on fundamental policy
questions [...] This experience illustrates the need for
flexibility in data collection, especially when policymakers
consider extending new policies or need to evaluate them in
real time for other reasons. Ideally, some sort of ‘rapid
response’ data gathering capacity.”
— Alan Krueger, “Stress Testing Economic Data”

7. “The collection of statistics needs to be modernized;
it is time to use the new technologies to start
collecting data.
…particularly important in developing countries
where the prevalence of mobile phones now offers
an unprecedented opportunity to measure the
economy.”
— Diane Coyle, “GDP”

9. OMGWTFGDP

10. “However, at this moment in survey research,
uncertainty reigns. Participation rates in
household surveys are declining throughout
the developed world. Surveys seeking high
response rates are experiencing crippling cost
inflation. Traditional sampling frames that
have been serviceable for decades are fraying
at the edges.”
— Robert Groves, “Three Eras of Survey
Research”

12. Orchestrating
Collective
Intelligence

14. PREMISE APP
Directed on-
the-ground
data acquisition

15. Crowdsourcing vs Orchestration

16. Crowdsourcing
survey

17. Crowdsourcing
survey survey tasks

18. Crowdsourcing
survey tasks workerssurvey

19. Orchestration
survey

20. Orchestration
survey survey tasks

21. Orchestration
survey tasks workerssurvey

22. Orchestration
survey tasks workerssurvey

23. Orchestration
survey tasks workerssurvey

24. Orchestration
survey tasks workerssurvey

25. Orchestration
survey tasks workerssurvey

26. survey campaign
allocation quality control
analytics
end-user
data contributor
User poses a question that is best
answered by via actual, on-the-
ground observation at scale.
Question is translated into
an internal “specification” of
the data points needed to
answer the question: type,
location, frequency,
coverage, etc.
Inventory of data points
automatically allocated to
data contributor pool, taking
into account budget, agent
profiles and geography.
Data points are dynamically
priced.
Contributors collect data in the
field using Android phones…
… which are sent back to the
Premise network.
QC is a mix of automated
(outlier detection; machine
learning; computer vision)
and manual (directed
sampling using oDesk)
checks.
Automated capabilities to
explore data and expose
trends or patterns;
hypothesize new features to
explain variation; suggest
specification refinement;
improve automated
verification.
end user
data
contributor
PLATFORM

27. Resource Scarcity
and
Access Risk

30. Average wait times are about ~10m longer in
Maracaibo than in Caracas.
Police are present ~80% of the time in
Maracaibo, but only 30-40% in Caracas.

31. Machine Learning

32. survey campaign
allocation quality control
analytics
end-user
data contributor
User poses a question that is best
answered by via actual, on-the-
ground observation at scale.
Question is translated into
an internal “specification” of
the data points needed to
answer the question: type,
location, frequency,
coverage, etc.
Inventory of data points
automatically allocated to
data contributor pool, taking
into account budget, agent
profiles and geography.
Data points are dynamically
priced.
Contributors collect data in the
field using Android phones…
… which are sent back to the
Premise network.
QC is a mix of automated
(outlier detection; machine
learning; computer vision)
and manual (directed
sampling using oDesk)
checks.
Automated capabilities to
explore data and expose
trends or patterns;
hypothesize new features to
explain variation; suggest
specification refinement;
improve automated
verification.
end user
data
contributor
PLATFORM

33. survey campaign
allocation quality control
analytics
end-user
data contributor
User poses a question that is best
answered by via actual, on-the-
ground observation at scale.
Question is translated into
an internal “specification” of
the data points needed to
answer the question: type,
location, frequency,
coverage, etc.
Inventory of data points
automatically allocated to
data contributor pool, taking
into account budget, agent
profiles and geography.
Data points are dynamically
priced.
Contributors collect data in the
field using Android phones…
… which are sent back to the
Premise network.
QC is a mix of automated
(outlier detection; machine
learning; computer vision)
and manual (directed
sampling using oDesk)
checks.
Automated capabilities to
explore data and expose
trends or patterns;
hypothesize new features to
explain variation; suggest
specification refinement;
improve automated
verification.
end user
data
contributor
allocation
PLATFORM

34. survey campaign
allocation quality control
analytics
end-user
data contributor
User poses a question that is best
answered by via actual, on-the-
ground observation at scale.
Question is translated into
an internal “specification” of
the data points needed to
answer the question: type,
location, frequency,
coverage, etc.
Inventory of data points
automatically allocated to
data contributor pool, taking
into account budget, agent
profiles and geography.
Data points are dynamically
priced.
Contributors collect data in the
field using Android phones…
… which are sent back to the
Premise network.
QC is a mix of automated
(outlier detection; machine
learning; computer vision)
and manual (directed
sampling using oDesk)
checks.
Automated capabilities to
explore data and expose
trends or patterns;
hypothesize new features to
explain variation; suggest
specification refinement;
improve automated
verification.
end user
data
contributor
analytics
PLATFORM

35. survey campaign
allocation quality control
analytics
end-user
data contributor
User poses a question that is best
answered by via actual, on-the-
ground observation at scale.
Question is translated into
an internal “specification” of
the data points needed to
answer the question: type,
location, frequency,
coverage, etc.
Inventory of data points
automatically allocated to
data contributor pool, taking
into account budget, agent
profiles and geography.
Data points are dynamically
priced.
Contributors collect data in the
field using Android phones…
… which are sent back to the
Premise network.
QC is a mix of automated
(outlier detection; machine
learning; computer vision)
and manual (directed
sampling using oDesk)
checks.
Automated capabilities to
explore data and expose
trends or patterns;
hypothesize new features to
explain variation; suggest
specification refinement;
improve automated
verification.
end user
data
contributor
quality control
PLATFORM

36. Optimizing Task
Allocation

37. TASKS

39. locations
measurables
CAMPAIGN DEFINITION

40. locations
measurables
CAMPAIGN DEFINITION

41. locations
measurables
CAMPAIGN DEFINITION

42. locations
measurables
CAMPAIGN DEFINITION

43. locations
measurables
CAMPAIGN DEFINITION

44. locations
measurables
survey period 1
CAMPAIGN DEFINITION

45. locations
measurables
CAMPAIGN DEFINITION
survey period 1 survey period 2

46. locations
measurables
survey period 2
CAMPAIGN DEFINITION
survey period 1

47. locations
measurables
survey period 1
TASK ALLOCATION
user 1
user 2
user 3
allocation period: 1

48. locations
measurables
survey period 1
TASK ALLOCATION
user 1
user 2
user 3
allocation period: 1 2

49. locations
measurables
survey period 1
TASK ALLOCATION
user 1
user 2
user 3
allocation period: 1 2 3

50. TASK COMPLETION RATE MODEL
payout
pTCR
“uptake risk”
Model features: user-history, task-history /
location-history, task-user, location-user
Issues: data sparsity in marginal vs
conditional, uptake counterfactuals (non-
iid sampling), path-dependence / lock-in
Linear functional model
}

51. Exploration
vs
Survey
Consistency

55. locations
measurables
period 1 period 2

56. locations
measurables
period 1 period 2

57. TASK REFINEMENT

58. ITERATIVE LOCATION DISCOVERY

59. Exploration vs Survey Consistency
- Campaign layers: separate discovery and survey
- Iteratively refine attribute and geospatial targeting
- Monitor correlation in item responses and
appearance of new attributes
- Monitor residual endogeneity

60. Fraud and
Coalition
Formation

62. Coalitions vs Referrals
- Referrals are necessary to reach most remote areas
- However we need to be able to partition the
Premise graph into independent subnetworks, e.g.
for re-evaluation, experimentation and sample
stratification.

63. CONTRIBUTOR AFFINITY MODEL
Model features:
direct referral
account features
upload location
visit histories
geographic area
response correlation
Issues: bootstrapping affinity scores for
new users, optimal scheduler is
antagonistic for coalition discovery
Sampling from Large Graphs [Leskovec & Faloutsos; 2006]
weight

64. RECAP
- Orchestrating collective
intelligence
- Optimizing task allocation via
dynamic scheduling and incentives
- Exploration and discovery while
maintaining survey consistency
- Fraud and coalition formation in
networks

65. QUESTIONS?
instagram/premisedata
(all images in this talk)
joe@premise.com | @josephreisinger

66. PROOF PROOFAUTO QC PROOFMANUAL QC MANUAL QCREVALIDATION

67. “The problem of changing statistics is that you lose the
ability to compare across time. The longer the time-
series, the harder it is to change it, but you want to be
able to compare. How do you replace GDP? And if you
do, you lose the past sixty years of relevance. This has
been a problem for centuries—take the Spanish silver
trade. Anything you measure will become increasingly
irrelevant over time.”
— Hans Rosling
[Zachary Karabell, The Leading Indicators]

70. “You need to focus on quality.
You’ll be better off with a
small but carefully structured
sample rather than a large
sloppy sample.”
— Hal Varian, Google

71. “Big Data is bullshit”
— Harper Reed

72. Big Data, n.: the belief that
any sufficiently large pile of
shit contains a pony with
probability approaching one
—@grimmelm

74. “dividing by bieber”