The traditional use of Bayesian classifiers is to have one period of learning followed by continuous application of the classifier to a much wider set of documents. However, continuous use of classifiers is also common when context grows over long periods of time. In the latter case, it is common to use classifier output as a recommendation which humans can accept as is or change when the classification is wrong. This paper proposes an interface that minimizes human interaction in such environments. The core concept is based on metromaps where documents are assigned to ``train lines'' first while the overall certainty of classification is based on documents' affiliation with multiple lines.

2. .
Learning = Social (software) Robotics
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 2/25
...
2/25

3. .
Social Robotics in Knowledge
Rebot
(careless)
Input
Human
Human
{structure}
(pinpoint)
Select
Browse
(or use otherwise)
Some
Knowledge
(folksonomies,
knowledge bases,
databases, indexes,
ontologies, etc.)
(metromaps )
07 myself+0 "On Context Management Using Metro Maps" SOCA, Matsue, Japan (2014)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 3/25
...
3/25

4. .
Metromap: The Basic Concept
07 myself+0 "On Context Management Using Metro Maps" SOCA, Matsue, Japan (2014)
14 K.Nesbitt+0 "Getting to more abstract places using the metro map metaphor" 8th IV (2004)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 4/25
...
4/25

5. .
A Practical Setting
Accident Something happened at Site A
Causes Part A, Part B, Part C, … Human Factors…
All Parts Part Z, Part Y, …, Human Manuals, …
Rating
Blackswan
scenario
management
platform
Storage,
Database
Human
judgment
Auto
judgement
Report
on site
07bmyself+0 "Black Swan Disaster Scenarios" PRMU研 (2014)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 5/25
...
5/25

6. .
Definitions, Objectives, Terminology
.
Different Viewpoint
..
.
classifier is not for finding hidden relations, but for clear separation
between known and new
.
Learning Classifier
..
.... a classifier that improves its inference over time based on human feedback
.
Metromaps
..
.... are used as the graphical interface between humans and robots
• MDC: Multi-Dimensional Classification
• MC: Metromap Classifier
• folksonomy: BigData with very frivolous management of metadata
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 6/25
...
6/25

7. .
Existing MDC Methods
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 7/25
...
7/25

8. .
MDC Basics: Binary Relevance (BR)
• binary: YES or NO for each Y 11
• problem: no relation between classes Y -- this is where metromaps can be
helpful
Training
Tuples
x1 x2 Y1 Y2 Y3
1 0.7 0.4 1 1 0
2 0.6 0.2 1 1 0
3 0.1 0.9 0 0 1
4 0.3 0.1 0 0 0
h1: X → Y1
h2: X → Y2
h3: X → Y3
11 J.Read+3 "Classifier chains for multi-label classification" Machine Learning, SpringerS (2011)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 8/25
...
8/25

9. .
MDC Basics: PairWise Sets (PW)
• relations can be found by creating new classes for all unique pairs in Y 11
• problem: many classes = fuzzy results = low reliability
Training
Tuples
x1 x2 Y1 Y2 Y3
1 0.7 0.4 1 1 0
2 0.6 0.2 1 1 0
3 0.1 0.9 0 0 1
4 0.3 0.1 0 0 0
h1: X → Z1
h2: X → Z2
Z1 Z2
1 0
0 1
0 0
0 0
11 J.Read+3 "Classifier chains for multi-label classification" Machine Learning, SpringerS (2011)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 9/25
...
9/25

10. .
MDC Basics: Label Combination (LC)
• basically, the extreme case of PW 11
• the same problem only worse -- there are too many classes!
Training
Tuples
x1 x2 Y1 Y2 Y3
1 0.7 0.4 1 1 0
2 0.6 0.2 1 1 0
3 0.1 0.9 0 0 1
4 0.3 0.1 0 0 0
h: X → Z
Z
1
0
0
0
11 J.Read+3 "Classifier chains for multi-label classification" Machine Learning, SpringerS (2011)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 10/25
...
10/25

11. .
MDC Basics: Classifier Chains (CC)
• classes are used in sequence 11
• merit: small number of classes -- only the necessary ones are used
• demerit: what is the correct order?
Training
Tuples
x1 x2 Y1 Y2 Y3
1 0.7 0.4 1 1 0
2 0.6 0.2 1 1 0
3 0.1 0.9 0 0 1
0.3 0.1 0 0 0
h1: X → Y1
h2: Y1 → Y2
h3: Y2 → Y3
h2h1 h3
4
11 J.Read+3 "Classifier chains for multi-label classification" Machine Learning, SpringerS (2011)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 11/25
...
11/25

12. .
MDC Basics: Graphical Methodology
• the graphical
methodology
behind MDC 03
• all about joint
probability and how
it is calculated using
graph theory
03
D.Koller+1 "Probabilistic Graphical
Models: Principles and Techniques"
MIT Press (2009)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 12/25
...
12/25

13. .
Metromap Classifier (MC)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 13/25
...
13/25

14. .
Metromap Classifier (before)
• problem: human load is too high!, ex: disaster scenarios 07b
Human
judgment
Auto
judgement
Folksonomy
07bmyself+0 "Black Swan Disaster Scenarios" PRMU研 (2014)
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 14/25
...
14/25

15. .
Metromap Classifier (after)
Human
judgment
Auto
judgement
Folksonomy
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 15/25
...
15/25

16. .
Metromap Classifier : Features
• human role
1. build the metromap = relations between classes
2. when robot fails, do the work manually
3. do the human part (by design) of the work
• robot role
1. classify incoming data into YES or NO for question: should human see
this?
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 16/25
...
16/25

17. .
Experiment
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 17/25
...
17/25

18. .
Experiment : Setup
• IEICE/ken is the source of data -- over 3000 presentations over 2-3 last
years
• various combinations of title, keywords, abstract
• usecase: which presentations should I look at closely?
◦ ... meaning the metromap reflects my personal research interests
• Dumb Classifier (DC): one-dimensional yes or no
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 18/25
...
18/25

19. .
Metromap Design: The Human
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 19/25
...
19/25

20. .
Metromap Classifier: Logic
• logic followed by the MC Robot
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 20/25
...
20/25

21. .
Results: Title only
0 20 40 60 80 100 120
Time sequence
0
10
20
30
40
50
60
70
80
90
Goodcount
Dumb ClassifierMetromap Classifier(smart) Hits on a timeline
title
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 21/25
...
21/25

22. .
Results: Title + Keywords
0 20 40 60 80 100
Time sequence
0
10
20
30
40
50
60
70
80Goodcount
Dumb ClassifierMetromap Classifier(smart) Hits on a timeline
title:keywords
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 22/25
...
22/25

23. .
Results: Title + Keywords + Abstract
0 20 40 60 80 100 120
Time sequence
0
10
20
30
40
50
60
70
80
90Goodcount
Dumb ClassifierMetromap Classifier(smart) Hits on a timeline
title:keywords:abstract
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 23/25
...
23/25

24. .
Wrapup: Not Good Enough
• not perfect: about 30% of wrong decisions
◦ FP: robot makes human look at bad stuff (false positive)
◦ FN: robot passes on good stuff (false negative)
• future improvements: need a solid logic which avoids FP and FN cases
• note: current naive and MDCs are at most 40-60% reliable -- no help here!
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 24/25
...
24/25

25. .
That’s all, thank you ...
M.Zhanikeev -- maratishe@gmail.com -- Metromaps for Minimizing Human Interaction with Bayesian Classifiers -- http://bit.do/marat150129 -- 25/25
...
25/25