1) An analysis of machine learning and human-analytics classification models that found human-guided models performed better due to incorporating "soft knowledge" unavailable to machine models. 2) Two case studies were conducted comparing decision trees from visual analytics with human guidance to those from standard machine learning algorithms. 3) Humans were able to leverage soft knowledge like imagining outliers, looking ahead to future decisions, and incorporating domain expertise to construct superior classification models.

1. An Analysis of Machine- and Human-
Analytics in Classification
Authors:
1. Gary K. L. Tam (Swansea University)
2. Vivek Kothari (University of Oxford)
3. Min Chen (University of Oxford)
Presented by:
Subhashis Hazarika
(Ohio State University)

2. Major Contribution
• An information-theoretic model that explains why a human driven visual
analytic model of classification performs better than a purely machine-
learning model.

3. Overview
• Consider two classification case studies.
• Create a decision tree classifier applying standard ML algorithms.
• Create a decision tree classifier using visual analytics guided by “soft
knowledge” of a human model-developer.[1]
• Using Information theory explain why the human centric approach
performs better than the ML approach.
• Quantify the “soft knowledge” that a human centric approach takes
advantage of.
[1]: “Visualization of Time-Series Data in Parameter Space for Understanding Facial Dynamics”, G.K.L. Tam, H. Fang, A. J.
Aubrey, P.W. Grant, D. Marshall, M. Chen. Eurovis2011.

4. Case Study A (Facial Dynamics Data)
• Input and feature extraction:
– 68 raw facial videos classified as one of the four (smile, sadness, surprise, anger).
– For each video, extracted 14 time series representing different temporal facial features.
– For each time series, 23 quantitative measures were obtained.
– Resulting in 14x23 attributes/features per video.
• Create Decision Tree using a Parallel Coordinates based visual analytics
system.
• Create a Decision Tree with standard ML algorithms (C4.5 or CART).

5. A: Visual Analytics Approach

6. A: Interactive Visualization

7. A: Outliers and anomalies

8. A: Building the D-tree

9. A: VA v/s ML

10. Case Study B (Visualization Image Classification)
• Input and feature extraction:
– 4x49 jpeg images classified as (bubble-chart, treemap, parallel-coordinate, bar-graphs).
– For each image, extracted 222 features via. different image classification and clustering .
• Create Decision Tree using a Parallel Coordinates based visual analytics
system.
• Create a Decision Tree with standard ML algorithms (C4.5 or CART).

11. B: Building the D-Tree

12. B: Comparative Evaluation

13. The Team
• Case Study A:
– Conducted by 7 researchers with expertise in vision, visual analytics, computer graphics
and machine learning.
– Human-centric D-Tree was constructed by a researcher who was specialized in graphics
and acquired the knowledge of computer vision and visual analytics during the project.
• Case Study B:
– Conducted by 2 researchers with expertise in image processing and visual analytics.
– Human-centric D-Tree was constructed by a researcher with 8 months of experience in
visual analytics.

14. But Why? Some Empirical Observations
• O1: Overview and Axis Distribution.
– A machine-centric approach examines many cut positions on all the axis and greedily
picks the cut with the highest quality measure.
– While a human model developer usually first obtains a general overview of the data and
identifies important axes with promising patterns before paying detailed attention to
these axes.
• O2: General Agreement amongst Statistics.
– ML algorithms only use one metric to determine the cut.
– HC approach can evaluate more than one statistics to decide the cut.
• O3: Look-ahead.
– Humans’ insights into the consequence often influences the current decision.
– Humans’ look-ahead ability enables multi-step judgement, while the ML algorithms
focused only on the current decisions.

15. But Why? Some Empirical Observations
• O4: Outliers.
– If possible model developers avoid axes with outliers, as they may be unreliable.
– Such reasoning is not available in the ML algorithms.
• O5: Cut Positions on an Axis.
– Humans’ look for a cut or cuts that would allow each class to expand beyond the current
instance in the training set.
– ML algorithms decide the cuts at the very edges of a particular class.
• O6: Human (Domain) Knowledge.
– Humans’ incorporate their domain knowledge into their model construction process.

16. Information Flow

17. Information Theoretic Analysis
• Estimated World Population : 7.4 billion
• Consider each person have 5 variations for each of the 4 expressions.
• The number of possible scenarios to capture : 148 billion
• The maximal entropy is 37.1 bits.
• We only know 68 cases(the raw training video)
• That is 1.7 x 10-8 bits. (a drop in the ocean)
• [ML] Optimistically, assuming the categorization retains 50% mutual
information. That leaves us with 8.5 x 10-9 bits of information.

18. Information Theoretic Analysis
• [VA] Model developer may know some 200 people reasonably well, and
can recall their 5 variations of 4 expressions at ease. Conservatively, that is
equivalent to 4068 videos instead of 68. Representing 1.0 x 10-6 bits of
known information.
• [VA]When given an arbitrary facial image, the developer can also
reconstruct an expression using imagination e.g at least 1 variation per
expression. This ability accounts to 29.6 billion videos, representing 7.4
bits of known information. (This ability shows up in determining outliers).
• 7.4 bits v/s 8.5 x 10-9 bits . That is roughly 871 million times more
information content.

19. Soft Knowledge and Soft Models
• Soft Knowledge: The uncaptured information not available to the
machine-centric approach.
• Soft Model: The models which make decisions based on soft knowledge.
• Examples:
1. Given a facial photo (input), imagine how the person would smile (output).
2. Given a video (input), determine if it is an outlier (output).
3. Given a set of points on an axis (input), decide how many cuts and where they are
(output).

20. Soft Models

21. Conclusion
• There is an overwhelming amount of information available to the human-
centric approach in the form of soft knowledge that can’t be utilized by a
machine-centric approach.
• It is necessary to understand and quantify the information flow in both
machine- and human-centric approaches to help design a mixed model
performing an much better job.
• Human model developer can never by cast aside.