Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
STRUCTURED LABELING
TO FACILITATE CONCEPT
EVOLUTION IN MACHINE
LEARNING
Presenter: Hillol Sarker
Authors
Todd Kulesza, Sal...
Motivation
 Machine Learning
 We want to train a machine according to some
target concept
 Supervised machine learning ...
Problem
 Labeling Consistency is compromised
 Labeler
 Expertise
 Familiarity with concept
 Judgment ability
 Data C...
Semantic Location
Concept EvolutionConcept Evolution
Introduction Preliminary Study Incorporate Feedback Study Result Conc...
Existing Approach
 Machine Learning approaches
 Noise-tolerant algorithm
 Multiple labeler
 Majority voting
 Weightin...
Approach
 Conduct series of formative studies
 In order to investigate concept evolution in
practice
 Observations and ...
Preliminary Study 1
 Researchers/practitioners create guidelines for
labelers
 Interviewed 2
 Feedbacks
 Guideline cre...
Preliminary Study 2
 Recruited 11 machine learning expert
 Binary choice task
 Prototype Software
Introduction Prelimin...
Preliminary Study 3
 Conducted on 9 of previous 11
participants 4 weekapart
 Using Same Prototype Software
 Same conten...
Incorporate Feedbacks in Study
Software
Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
Study Software Interface
 Experiment tested 3 interface conditions
 Baseline
 Traditional Mutually Exclusive “Yes”, “No...
Study Procedure
 15 participant
 108 items to label
 Fixed task order
 Cooking, travel,
and gardening
 Study Procedur...
Result: Group
Group Count
Structured > Baseline (p<0.001)
 Manual > Baseline (p<0.001)
 Assisted > Baseline (p<0.001)
P...
Result: Revision
Revisited Count
 Manual > Baseline (p<0.005)
 Assisted > Baseline (p<0.005)
Revised Count
Structured >...
Result: Label Quality
 Matric ARI (Adjusted Rand Index)
 Measures Agreement
 Pairs of items that should end up together...
Result: Labeling
 Labeling Speed
 Manual < Baseline (p=0.003)
 Assisted < Baseline (p<0.001)
Introduction Preliminary S...
Feedback
 Participant ranked
each tool as their
favorite
 Ho w o fte n did yo ur
concept change?
 Likert-scale
Favorite...
Summary
 Structured Labeling
 Helps people evolve concept
 Increases label consistency
 at cost of speed
 Can help Ma...
Contribution
 Concept evolution causes inconsistent
labeling
 Being first to show its importance
Not Significant Differe...
Critique of work
 Fixed task order used
 e.g., Cooking, travel, and gardening
 Carry over effect
 Limited to supervise...
Upcoming SlideShare
Loading in …5
×

2014.chi.structured labeling to facilitate concept evolution in machine learning

Paper presentation

  • Login to see the comments

  • Be the first to like this

2014.chi.structured labeling to facilitate concept evolution in machine learning

  1. 1. STRUCTURED LABELING TO FACILITATE CONCEPT EVOLUTION IN MACHINE LEARNING Presenter: Hillol Sarker Authors Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
  2. 2. Motivation  Machine Learning  We want to train a machine according to some target concept  Supervised machine learning needs consistent labeled data  e.g., spam filter, email prioritize  Difficult to obtain Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  3. 3. Problem  Labeling Consistency is compromised  Labeler  Expertise  Familiarity with concept  Judgment ability  Data Contains  Ambiguity  Changing distribution  Concept change over time Example?Example? Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  4. 4. Semantic Location Concept EvolutionConcept Evolution Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  5. 5. Existing Approach  Machine Learning approaches  Noise-tolerant algorithm  Multiple labeler  Majority voting  Weighting scheme  Pairwise comparison (A better fit, then B)  Problem: No human judgment Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  6. 6. Approach  Conduct series of formative studies  In order to investigate concept evolution in practice  Observations and feedbackfrom these studies informed final prototype  Incorporate feedbacks on initial labeler software  Design a Study  Evaluate proposed Structured Labeling Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  7. 7. Preliminary Study 1  Researchers/practitioners create guidelines for labelers  Interviewed 2  Feedbacks  Guideline creation process is iterative  Evolves observing new data  e.g., examples with multiple interpretation Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  8. 8. Preliminary Study 2  Recruited 11 machine learning expert  Binary choice task  Prototype Software Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  9. 9. Preliminary Study 3  Conducted on 9 of previous 11 participants 4 weekapart  Using Same Prototype Software  Same content but shuffled order Not Significant Difference Significant Difference Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  10. 10. Incorporate Feedbacks in Study Software Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  11. 11. Study Software Interface  Experiment tested 3 interface conditions  Baseline  Traditional Mutually Exclusive “Yes”, “No”, “Could be”  Structured  Manual Structuring  Structured Labeling  Assisted Structuring  Structured Labeling + Automated Assistance Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  12. 12. Study Procedure  15 participant  108 items to label  Fixed task order  Cooking, travel, and gardening  Study Procedure  Brief Introduction  Time to practice  Log interaction in each interface  Completion of each task=>Questionnaire  Completion of 3 task=>Questionnaire Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  13. 13. Result: Group Group Count Structured > Baseline (p<0.001)  Manual > Baseline (p<0.001)  Assisted > Baseline (p<0.001) Pages perGroup Could be < Yes or No Yes < No No Could Be Yes Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  14. 14. Result: Revision Revisited Count  Manual > Baseline (p<0.005)  Assisted > Baseline (p<0.005) Revised Count Structured > Baseline (p<0.011)  Manual > Baseline (p<0.006)  Assisted > Baseline (p<0.024) First Half Last Half Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  15. 15. Result: Label Quality  Matric ARI (Adjusted Rand Index)  Measures Agreement  Pairs of items that should end up together over all possible pairs  Label Quality  Manual > Baseline (p=0.02)  Assisted > Baseline (p=0.02)  Manual ≠ Assisted (P=0.394) Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  16. 16. Result: Labeling  Labeling Speed  Manual < Baseline (p=0.003)  Assisted < Baseline (p<0.001) Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  17. 17. Feedback  Participant ranked each tool as their favorite  Ho w o fte n did yo ur concept change?  Likert-scale Favorite Lease Favorite Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  18. 18. Summary  Structured Labeling  Helps people evolve concept  Increases label consistency  at cost of speed  Can help Machine learning algorithm  Weight forgroups (e.g., “definitely yes” vs. “yes”) Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  19. 19. Contribution  Concept evolution causes inconsistent labeling  Being first to show its importance Not Significant Difference Significant Difference Introduction Preliminary Study Incorporate Feedback Study Result Conclusion
  20. 20. Critique of work  Fixed task order used  e.g., Cooking, travel, and gardening  Carry over effect  Limited to supervised learning  Assisted structuring  Not always possible  May bias decision Introduction Preliminary Study Incorporate Feedback Study Result Conclusion Thank You

×