This is a summary of the latest research on model interpretability, including Recurrent neural networks (RNN) for Natural Language Processing (NLP) in terms of what's in an RNN. In addition, it contains suggestion to improve machine learning based user interface, to engage users and encourage them to contribute data to adapt the models to them.

1. Bridging the gap
between AI and UI

2. ● Recommendation Systems (RecSys)
● Model Interpretability Technics (MI)
● Using MI for communication
Outline

3. Recommendation
Systems

4. Netflix RecSys

5. What does it mean?

7. RecSys with Deep Learning
Autoencoders Meet Collaborative Filtering (Sedhain et al. AutoRec 2015)
Session Based Recommendations with RNN (Hidasi et al. ICLR 2016)
Wide & Deep Learning for Recommender Systems (Cheng et al. DLRS @ RecSys 2016)

8. Matching Engine

9. Response Suggestions
Not what I wanted to say!

10. UI for AI

11. Better UI for AI

12. Google Apple

13. Google Apple
Search suggestions (auto-complete) moves to search results

14. Feedback
- Constructive
- Focused

15. Understanding
Models
Predictions

16. How to “understand” a model prediction?

17. How to explain a model prediction?
Which features contributed to the
model prediction?
And how?
Example: Decision Tree

18. “Why should I trust you?” - Explaining the Predictions of
Any Classifier
LIME: Local Interpretable Model-Agnostic Explanations

19. LIME
- Explains the model result
- Enhances User Trust
Supports Multi-class classifications, (example),
text documents, images, etc.
Packages exists for R, Python, etc.

20. LIME
● Linear approximated model with a subset of the features - O(2n)
● Lowest distance to the original inputs
● Measuring the magnitude of the predictions distance

21. SHAP - NIPS`17
SHapley Additive exPlanations
“A Unified Approach to Interpreting
Model Predictions” - Lundberg & Lee
“Connects game theory
with local explanations”
SHAP Value - Feature importance as
an impact (effect) on the output

22. - The marginal value of an agent in a coalition (impact)
- Average marginal contribution over all possible sequences
Cooperative Games:
http://www.lamsade.dauphine.fr/~airiau/Teaching/CoopGames/2011/coopgam
es-7[8up].pdf
Game Theory :: Shapley Values

23. SHAP - NIPS`17
- Calculates N linear models for a subsets of the features
- Calculates the impact on the result
- How much each feature contribute as part of a ‘coalition’?

24. SHAP - NIPS`17
Unifies:
- LIME
- Shapley sampling values
- Shapley regression values
- DeepLIFT
.
.
.
Better consistency with
human intuition

25. Deeper
Understanding
of Deep Learning?

26. Visualizing Image Classifications
As an interpretability method:
● What features are these networks really using?
● Do individual units have meaning?
● What roles are played by different layers?
● How are high-level concepts built from low-level ones?

27. Visualizing Image Classifications
Network Dissection: Quantifying Interpretability of Deep Visual Representations
https://arxiv.org/pdf/1704.05796.pdf
http://people.csail.mit.edu/bzhou/ppt/presentation_ICML_workshop.pdf

28. Deeper
Understanding
Of Deep Learning
for NLP

29. Why?
< 2014 Rule-based NLP
2014 - DL NLP
Future DL identifies NLP Rules
= Transparent
= Black box

30. How?
CBOW (Word-vectors)
LSTM / BiLSTM
AutoEncoders

31. What is encoded / captured in a vector?
Fine-grained Analysis Of Sentence Embeddings Using Auxiliary Prediction Tasks
(Adi et al. ICLR 2017)
● Trained a model to predict: sentence length, word existence, word-order
● 300-D CBOW - most effective (even for word order!)
● LSTM Autoencoder (500 to 750 D)

32. What is encoded / captured in a vector?
Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive
neural networks process hierarchical structure
(Hypkes et al. NIPS 2016)
● Visualizing activated neurons
● Researching compositional structures (trees)
● “The scientist who wrote the natural language research paper”

33. Analyzing Hidden Representations in End-to-End Automatic Speech Recognition
Systems
(Belinkov, Glass, NIPS 2017)
● Speech Recognition
● Visualizing layers
● Which layer/neuron is responsible to which phone (sound)
What is encoded / captured in a vector?

34. What you can cram into a single $&!#* vector:
Probing sentence embeddings for linguistic properties
(Conneau et al, ACL 2018)
● Ray Mooney
● Set of tests:
○ Surface information: word content, sentence length…
○ Syntactic Information: sentence ‘correctness’, hierarchical structure (depth)
○ Semantic Information (tense, word usage)
● Testing the information that is captured in different vectors
● Again CBOW and BiLSTM stars
What is encoded / captured in a vector?

35. RNN - Debugging Translation
http://seq2seq-vis.io/ (Strobelt, 2018)
Examines the 5 stages (Encoder, Decoder, Attention, Predictions, Beam-search)
Nearest Neighbor
Visualization
- Examine model decisions
- Connect decisions to
previous examples
- Test alternative decisions

36. More open questions
What linguistic structures can be captured by
RNN?
How does a model reach a decision?
When would a model fail?
What can’t the model do?
http://u.cs.biu.ac.il/~yogo/blackbox2018.pdf

37. Keep-Current

38. Activities - Keep-Current Meetup
- Learning by Doing
- Machine Learning Seminar - Monthly in Vienna
- Applied Machine Learning Course
- PyTorch, Fast.AI

39. Keep-Current
Educational Project
Filter newly released academic papers
Hybrid recommendation system
- Collaborative filtering
- Content Similarity
- User-Feedback (MOT)
- RNN

40. Methods
Topic extraction (LDA / LSI)
Document Vector Representation
- Citation Correlation Matrix Decomposition
- Paragraph Vectors
- ELMo / BERT - Attention layer
Similarity using Spotify ANNOY (Approximate Nearest Neighbours - Oh Yeah!)

41. User-Input
Key-words / topics Extraction
Semantic Similarity
Vector distance:
- cosine similarity
- Euclidean Distance
- Word Mover’s Distance
- PIP

42. NIPS 18
PIP - Pairwise Inner Product

43. “Augmented Intelligence”
(Simon Stiebellehner)
“Human Teaching”
(Shai Herz)

44. Human-Machine
Interactions

45. Human-Machine Interactions - Ofra Amir
PhD, Harvard University
Intelligent Interactive Systems
(advanced topics in information systems)
- Technion, Israel
AAMAS'18:
- HIGHLIGHTS: Summarizing Agent Behavior to
People
- Agent Strategy Summarization

47. Meh...

48. Explain results
- Keywords Contribution:
Positive / Negative
- Encourage User Feedback
Model adaptation
- Stronger Engagement

49. Interpretability & Communication
http://yannickassogba.info/

50. Explainable AI
- Models are prune to make mistakes
- Interpretability to the rescue!
- Supplies a peek into the features
- Enhance user trust
- Enables a constructive feedback

51. Explainable AI using UI
- Be Creative: Explainable UI
- Using ‘stronger’ features for personalization
Future Research:
- Effective Agent Strategy Summary
- Expected Behavior under Different Conditions

52. Thank You
Liad Magen
www.linkedin.com/in/liadmagen
www.github.com/keep-current
Keep-Current Meetup
Liad.magen @ gmail.com