The document discusses hybrid machine learning and crowdsourcing approaches for various tasks. It proposes combining machine learning algorithms with human input to contextualize data, identify patterns, process difficult data, and improve algorithms. Example application areas mentioned include transportation (predicting congestion), shopping/advertising (understanding consumer behavior), healthcare (monitoring epidemics), and document translation. It also addresses challenges like privacy, cost, and ensuring high quality contributions from humans. Methods developed so far include crowdsourcing mobile app architectures, learning privacy policies with human guidance, combining Twitter data with transportation agency reports, and algorithms for estimating ground truth and contributor trustworthiness.

1. Taking the Pulse of the City
Next Generation Hybrid Learning and Analysis
Emil Lupu
Imperial College London
e.c.lupu@imperial.ac.uk
1©Emil Lupu
RISSwww.rissgroup.org

2. Taking the pulse of the city …
2

3. Improve users quality of life whilst simultaneously optimise
resource utilisation.
3

4. • Many of the human parameters are subjective and relate to
emotional state: happiness, fear, anxiety, optimism, security.
• Much of the data has a privacy dimension.
• Resource utilisation data does not capture user rationale and
intent.
– E.g. purpose of a trip on public transport vs Origin-Destination
• Aspects such as salience and meaning cannot easily be
captured through sensory data
– e.g., turn right when you reach a beautiful Edwardian house.
• Much of the on-body sensing is captured without context.
4

5. Extracting Knowledge from Data
• Analytics, typically assume that:
– The semantics of the data is known.
– The questions that we ask of the data are known.
• Machine Learning algorithms can uncover new patterns and
semantics from data but:
– Require data appropriately labeled.
– Require tuning and refinement iterations with feedback.
5

6. Combining Machine Learning with
Human contributions!
• Crowdsourcing of subjective
parameters.
• Contextualise data.
• Use user expertise in
identifying and recognizing
patterns.
• Process data that is difficult
to process automatically.
• Ask users to improve
algorithms.
• Human input to improve
machine inference.
• Use machine inference to
monitor and improve user
performance.
6

7. Application Areas
Transport
• Determine and predict congestion.
• Patterns and purpose of travel.
• Characterise congestion areas.
• Traveller preferences and why?
Shopping and Advertisement
• Why do users buy what they buy?
• Onset of fashions
• Products development
• Targeted advertisement and influence
7

8. Primary Healthcare
• Impact of public awareness campaigns
• Use of GP vs use of Emergency Care
• Spread of epidemics e.g., influenza
• Correlation of Symptoms
Pervasive Care
• Link sensed values to activities.
• Link sensed values to symptoms felt by
patient.
• Feedback to patient and improved
compliance.
Document Translation
• Improve machine translation with
idiomatic expressions
• Monolingual translation (cf. Benderson)
8

9. Privacy
• Learning Privacy and Information
Sharing Rules.
• Evaluate social context influence.
Image Processing
• Image recognition and improved
human performance in data
labeling.
• Emotion and Body language
recognition
• Salience and landmark based
navigation
• Crowdsourcing navigation
paths in busy environments
… and many others
9

10. Complex Integration
Workflows
Sensing (+labelling)
data
labelling + outlier removal
feature computation
candidate features, param
feature, param selection
learningdisambiguation
outcome
contributor selection
performance feedback
outcome, correction
10

11. Humans are Error Prone
• Lack of experience, knowledge
• Lack of care, due diligence.
• Over confidence.
• Cognitive bias.
• Availability heuristics.
• Cognitive vs motor skills. (cf. Ipeirotis)
• Crowdsourcing assumes that Information Quality can be
achieved with numerous contributions e.g., Wikipedia.
11

12. Crowdsourcing has a cost
• Financial (e.g., Mechanical Turk)
• Good will (e.g., Translation)
• User’s time
• Incentivisation (e.g., games)
• How to achieve desired information
quality at minimum cost?
• QoI and Contributor Trustworthiness
is a co-dependent process.
– QoI depends upon the trustworthiness
of the contributors.
– An annotator’s trustworthiness depends
upon the quality of his past work.
12

13. Engineering Hybrid Learning
• How to structure workflows that interweave
Machine Learning and human input?
• How to link QoI to number of contributions?
• How to determine trustworthiness of users?
• How to improve human input?
• How to rapidly deploy crowdsourcing tasks?
• How to best allocate tasks?
• How to predict time to answer?
• How to resist malicious use and collusions?
13

14. SOME OF THE THINGS WE HAVE
DONE SO FAR
14

15. A generic architecture for deploying
crowdsourcing tasks on Android
• Deploy crowdsourcing
tasks with minimal effort.
• Integrated data capture
from mobile phone
sensors.
• Automated scheduling of
tasks and sensor capture.
• Data upload to server with
meta data.
• Notification infrastructure
• Integration with ML tasks
34 CHAPTER 3. DESIGN
Persistence
System
Plugin ComponentsUser Interface
Task Store Activity View Task Activity
Infrastructure
Application Layer
Task Manager
Service
framework IPC
Handlers
Plugin
Applicaitons
Third-party
Plugin Activities
Third-party
Plugin Services
Notification
Service
Server
Communicator
Notification Receiver
Android
Legend:
Framework
Plugin
Data Provider
Database
DAOs
Notification
Channel
Android File Storage
Meta Content
Provider
SQL DatabaseNetwork Modules
Access
Control
Serialization/Deser.
Context
Service
Figure 3.2: Client Architecture
The User Interface layer contains only visual representation meant for the user. It does not
contain any functionality other than what’s needed for interacting with lower layers and presenting
information to the user. The user interacts with the UI to see, for example, what the available
tasks are, view their descriptions and obtain information on the tasks’ progress.
The framework’s main funcitonality is in the layers beneath the UI layer. The Task Manager
is the service that orchestrates behind the scenes the entire task execution and acts as a mediator
between other components.
There are several modules meant for creating, accessing and modifying both meta informa-
tion and generated data. The data model used by the modules has been discussed separately in
§3.2. In terms of the modules’ purpose, they provide di↵erent levels of storage abstraction in or-
3.5. CLIENT 37
defined purpose and the singleton is the logical entity that embeds all the functionality needed to
achieve this purpose. Concretely, this means that each and every one part of the plugin application
is actively working towards the fulfillment of this one goal; all of these components are children
of the stage application singleton in the plugin’s logical hierarchy. They should only exist as
long as the singleton exists, and should only work towards furthering its progress. Note that, as
shown on the Figure 3.2, the application singleton is shown to be part of the framework. This
is so because it is essential for the control and information exchange to happen according to the
well-defined communication protocol and therefore the implementation should be provided by the
framework. However, the information provision of the stage’s progress, how stage progress is
internally coordinated between the plugin’s components (for example, exposed global state or an
internal interface), and how some of the control orders are passed along to these components should
be left to the plugin developer.
The provided IPC protocol facilitates control and information exchange. Its implementation
comprises of several parts and the overall design is presented in Figure 3.3.
MCSStageApp
<Plugin> App
Framework Library Third-Party ApplicationFramework Application
<Plugin>
Main Activity
<Plugin>
Other
Components
..
Task Manager
Messenger
HandlerApp Messengers
IPC DefinitionTask Manager
IPC Definition
Handler
Legend:
Control
Information and Requests
Plugin-specific
Note: Framework can start and kill
the entire plugin application so while
the sub-controls are plugin-specific,
ultimately they can be overruled
Figure 3.3: Role of the provided library in the control and information exchange between the
15

16. • Data uploat
42 CHAPTER 3. DESIGN
Server
REST Resources
Web Client
REST API
Smartphone
Client
Tasks
Stage Data
Client
Notification
ML
Static HTML
and
Javascript
Participant
Meta
Specifications
Persistence &
Infrastructure
Meta DAO
ML Box
Blob DAO
Task DAO
External
Notification
Server
Partiicipant
DAO
File
System
(Blobs)
NoSQL
Google
Notification
Server
Figure 3.4: Server Architecture
The Meta Specifications is the resource representing the description of plugin stages.
This includes the required and optionally accepted parameters needed for the execution of the
described plugin. The resource also contains the unique name of the Android Intent action
that the framework application should use to invoke the plugin. The resource is to be used by
third-party developers when registering their new plugin. It is also to be used by the task creation
client through which new tasks are to be submitted. The client’s UI can be dynamically populated
depending on the stages the user selects to specify in the new task. The client can then locally
verify that the provided input parameters meet the plugins’ requirements. If the test passes, then
the representation of a Task resource is sent via POST to the Tasks resource.
The Tasks resource is a collection of children Task resources. It represents all the Generalized
Task meta specifications that have been input in the system. Upon receiving the POST request
from the task creation client, the Tasks resource also verifies the input after which it creates a full
49
sk Store and View Task screens.
milliseconds) should its value be recorded, and for how long (seconds) should there
us recording. These are all specified upon task creation through the web interface
task server resource. So is the case with all the parameters for the other plugins
t.
Figure 4.9: Actionable notification of an ongoing sensor collection.
n requests one file per sensor from the framework with the provided file request API.
at the readings are stored and later can be used by other stages. It has an optional
y that displays the collected data. The UI also allows to start and stop collection
also be done through the Notification shown to the user as shown on Figure 4.9. The
apture showcases the framework’s ability to support plugins that request files to
run in the background while having a UI with which the background process is tied
oreground running notification also controlling the background service.
a Upload
oadData plugin stage has the purpose of taking the produced data from any other
oading it to the framework’s server. The ids of the stages is the required input for
he plugin also allows to specify an alternative URL to which to perform the upload.
ust point to a service supporting a multiform POST HTTP request. This is used
plugin does not know in advance how many output files there are, nor their MIME
are established dynamically which provides the flexibility of being able to upload any
s of any kind. The POST is done over a secure connection and also contains the user
which helps protect their privacy since the data stored on the server is grouped per
y permission that the plugin uses is the Internet. It does not even require access to
se permissions are visible to the user and make it clear what the plugin does. It is
ell-defined purpose that is intended for the framework’s plugins. A user can see the
the plugin and decide if its functionality matches the permissions the application
n runs entirely in the background, only showing a notification of its progress to the
same progress that is also reported to the main application upon changes, and hence,
ored from the framework’s application, too.
t Labeling
50 CHAPTER 4. IMPLEMENTATION
Figure 4.2: Visual and Textual progress feedback.
16

17. Learning Privacy Policies with
Human Input
• Call admission. When does a user
accept or reject calls based on
CLID, time of day, location,
sensors, devices nearby?
• Inductive Logic Programming
– Preserve user familiarity, bias to learn
minimally differing rules.
• User can
– Review and edit rules.
– Select terms to be considered.
– Lock rules from subsequent revision
Joint work with Dr. A. Russo
17

18. Crowdsourcing, Twitter & TfL data
• Transport for London (TfL) distributes
data regarding traffic disruptions. Each
event defines location and severity.
• However,
– There is a delay before events are reported.
– The “area affected” is not characterised.
18

19. • Mining Twitter data for traffic
related information.
• Correlating twitter inferred data
with TfL reporting.
• Offering users ability to see “jam-
cams” near areas of disruption.
witter for local tra c prediction September 3, 2012
Figure 43: The density of tra c tweets with the EM.
Density of Tweets
Mining Twitter for local tra c prediction September 3, 2012
Figure 43: The density of tra c tweets with the EM.
Figure 44: The TfL disruptions for this time (10:00).
Again, the second method using the EM algorithm shows a more detailed view of tra c in
London, which corresponds to the TfL disruptions too.
5.6.9 Conclusion of the HeatMap representation
TfL reports
19

20. • Allow users to report additional
information.
Not investigated:
• Asking users in surrounding areas.
• Characterising congestions specific to
local areas with help from local
contributors.
• Incentivisation models for participation.
• Integration with NLP for Twitter (in
discussion with Sheffield Univ.)
20

21. Ground truth and trustworthiness
• For most crowdsourced processes:
– Ground truth is not known in advance.
– Human input is error prone.
• Example: Binary classification
– What is the correct label?
– Who is the more reliable?
0.0 0.2 0.4 0.6 0.8 1.0
specificity (true -ve rate)
0.0
0.2
0.4
0.6
0.8
1.0
sensitivity(true+verate)
5
10
15
20
25
30
35
40
45
num.labels
Reviewer ratings over
several years for a set of
conferences sharing many
PC members.
21

22. Existing approaches
• Majority voting.
• ML ests. of ground-truth and
reliability from observed labels.
(Dawid and Skene, 1979).
• MAP ests. of ground-truth and
reliability. Train classifier
(regresser) for ground-truth (Raykar
et al., 2009).
• Learn to blacklist poor annotators
from future labelling (Welinder and
Perona, 2010).
• Infer difficulty of individual items
(Whitehill et al., 2009).
I Objects, O
I Annotators, A
I Semi-trusted labels, L
I Estimate: ground-truths, zi
and reliabilities, aj
i 2 O
zi
(i, j) 2 L
li,j
j 2 A
aj
⇣ ↵
Simple probabilistic model of trust
0
0.0
0.2
0.4
0.6
0.8
sensitivity(true+verat
S
Raykar et al 2009
• Implementations of Raykar
et. al, 2009, and Wellinder
and Perona, 2010
algorithms.
• Used on several synthetic
and real data sets
22

23. Transferrable skill and a variational
approach to trustworthiness
• Different tasks have different
characteristics, difficulty,
knowledge required, etc.
• Some people are better at
some tasks.
• Some people are better at a
broader range of tasks.
• Skill vs. Knowledge vs. Diligence
• Can we predict performance as
task changes?
23

24. Experiments on Synthetic Data
Experiments on Synthetic Data
Scale Objects Scale Annotators Scale Tasks
24

25. Selecting annotators
• Labeling large data sets
• Marked improvements in
accuracy.
• Dynamic data acquisition
• Reduces number of labels
required.
• Reduce no of low quality
labels
Pairing Metrics
• Mutual Information
• Information Gain
• Thresholds
Integration of learning
with staging mechanism
25

26. Evaluating Cognitive vs. Motor Skills
(work in progress)
• Facial emotion recognition
data set.
• Recruit and track
performance of labellers.
• How do annotators improve
with experience.
• Frustrate, use of motor skills
as performance decreases.
• Combine with gamification;
incentives for accuracy.
• Combine with trust metrics.
acquisition of skills
peak performance
baseline motor skills
loss of
interest
26

27. Experiments in Gamification
• To use images for navigation it is necessary to pinpoint
accurately the location photographed and the place of the
photographer
27

28. References
• Luke Dickens, Emil Lupu:On efficient meta-data collection for
crowdsensing. PerCom Workshops 2014: 62-67
• Domenico Corapi, Alessandra Russo, Emil Lupu:Inductive Logic
Programming in Answer Set Programming. ILP 2011: 91-97
• Domenico Corapi, Oliver Ray, Alessandra Russo, Arosha K.
Bandara, Emil C. Lupu:Learning Rules from User Behaviour.
AIAI 2009: 459-468
28

29. 29