Presentation for automated labeling

1. Framework for semi-automated labeling for ground
truth residing in text allowing interactive expert
feedback
Tapan Shah
April 28, 2018

2. Outline
1. Introduction
2. Problem Motivation
3. Formulation and First-cut solution
4. Feedback types
5. Metrics and discussion

3. Introduction
1. Lead Scientist, GE Global Research
2. Interests: Time Series Analytics for PHM, massive imbalance in
large class problems, perception metrics, human feedback for model
improvement
3. PhD in signal processing from Tata Institute of Fundamental
Research
4. Thesis: Signal Processing for Low Precision quantization
5. B.Tech in Electronics from NIT Surat

4. Application
Figure: Schematic of workflow

5. Problem Motivation
1. We formulate a supervised multi-class classification problem for
automated troubleshooting
2. For part replacements, extracting label easy.
3. For non-part actions (reseating, adjustments, removals etc), not
obvious
4. Initially, we used expert-guided rules
4.1 Not scalable in terms of time and resources
4.2 No structure to conversation
4.3 Myopic view when assigning labels

6. Example
Service Request # Repair Action
1-198428217869 adjusted the linkage in the electri-
cal dock Contacted customer and
confirmed the system is down due
to electronic dock failure. Paged
FE for follow up.
1-186410196607 Assisted FE Pigg remove dock
and repairing broken wire going to
dock motor. found the switch con-
nector inside the table dock dis-
connected. reconnected the dock
switch.
Table: Examples of two Service Requests with similar "issues"

7. Our formulation
1. Formulated the label generation and assignment as Topic Modeling
problem
2. The topic modeling
2.1 Assigns each text document to a topic
2.2 Gives a set of important keywords characterizing each topic
2.3 Provides an initial seed point and a structure to communicate with
experts

8. Non-negative Matrix Factorization
1. Determine two matrices W ∈ Rn×k
and H ∈ Rk×m
such that
A ≈ WH.
2. These matrices are found by solving the following optimization
problem 1
:
min
W>0,H>0
||A − WH||F ,
3. Solved using alternative non-negative least squares method (NNLS).
W ← arg min
W>0
||A − WH||F , H ← arg min
H>0
||A − WH||F . (1)
1l1/l2 Regularization terms can be added to control sparsity

9. NMF for topic modeling

10. NMF for topic modeling
Figure: Toy example showing use of NMF for topic modeling

11. Identifying number of topics
1. There are topic coherence based metrics [NLGB10], [AGH+
13] to
measure the performance for the topic modeling.
2. We found those metrics not suitable for our application. We define a
new metric ReconK
reconK = ||A − WH||2
F + λ log k
where λ is σmin/σmax
3. Plot reconK as a function of k and chose the k corresponding to the
knee point

12. Incorporating expert feedback: Prior Work
1. The authors in [YPL+
14] allows for 2-3 types of user input for LDA
based topic modeling
2. The work closest to ours is [CLRP13]. The authors define 5 types of
user input and optimize a metric such that the updated W and H
matrices are close to the user feedback output

13. Incorporating expert feedback
Feedback type Description Mathematical update
Addition The expert is asked if all the ma-
jor "issues" or topics are covered.
If not, he is asked to provide them
along with associated keywords
Add a new column to W with 1s at
the location of corresponding key-
words.
Deletion The expert is asked if any of the
label is unimportant or unnecessary.
Delete the column in W.
Rename The primitive labels are renamed to
make them intuitive and informa-
tive
Create a simple mapping function
which maps the primitive label to
expert label.
Keyword modifi-
cation
For each label, the expert can re-
weigh the importance of the key-
words associated with that label.
The re-weigh can be binary or soft
scoring
For each column in the matrix W,
the corresponding weight of the
word that was re-weighed is mod-
ified.
Merging The expert is asked if two or more
labels are similar and can be merged
into a single label
In matrix W, the two columns to be
merged are removed replaced by a
single column which is the weighted
sum of the two deleted columns.
Splitting The expert can suggest if any par-
ticular label is too generic and
should be split into multiple labels.
If yes, those labels along with the
associated keywords for each split
label are recorded.
The corresponding column is re-
moved and replaced by two columns
with 1s at the locations of the cor-
responding keywords.
Table: Formal mechanism for expert feedback

14. Modified objective function to incorporate feedback
1. In order to maintain continuity and coherence, we do not want the
original matrix W to change a lot (This is very important from a
practical standpoint)
min
W>0,H>0
||A−WH||2
F +β1||W −Wfeedback ||2
F +β2||Wreduced −Wold ||2
F ,

15. Metric
1. We share a spreadsheet with text and the associated keyword and
label (Label is most imp noun and verb in keyword set) with >2
experts.
2. The expert is asked to provide feedback in a structure as defined
above.
3. We compute the disagreement index among the experts and
machine generated labels.
disagreement(j, k) =
1
n
i
I(expertj = expertk )
4. Comparing maxk disagreement(j, k) for j = expertmachine with
minj,k disagreement(j, k) for expertj , expertk ∈ experts.

16. Discussion
1. We got a maximum machine disagreement of 0.4 compared to
minimum expert disagreement of 0.25
2. Post-feedback, the machine disagreement was reduced to 0.3

17. Sanjeev Arora, Rong Ge, Yonatan Halpern, David Mimno, Ankur
Moitra, David Sontag, Yichen Wu, and Michael Zhu.
A practical algorithm for topic modeling with provable guarantees.
In International Conference on Machine Learning, pages 280–288,
2013.
Jaegul Choo, Changhyun Lee, Chandan K Reddy, and Haesun Park.
Utopian: User-driven topic modeling based on interactive
nonnegative matrix factorization.
IEEE transactions on visualization and computer graphics,
19(12):1992–2001, 2013.
David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin.
Automatic evaluation of topic coherence.
In Human Language Technologies: The 2010 Annual Conference of
the North American Chapter of the Association for Computational
Linguistics, pages 100–108. Association for Computational
Linguistics, 2010.
Yi Yang, Shimei Pan, Jie Lu, Mercan Topkara, and Doug Downey.
Incorporating user input with topic modeling.
In CIKM 2014 Workshop on Interactive Mining for Big Data
(ImBig’14), 2014.