Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
ActiVis
Visual Exploration of Industry-Scale
Deep Neural Network Models
Minsuk Kahng Pierre Andrews Aditya Kalro Polo Chau...
Deep Learning is Powerful
2Image credit:http://www.nvidia.com/object/drive-px.html, https://venturebeat.com/2016/04/14/, h...
Understanding Deep Learning is Challenging
3
Cat
Dog
Cat
Dog
INPUT OUTPUTMODEL
Image credit: https://www.kaggle.com/c/dogs...
Cat
Dog
Visualizing Instance Activation
4
Which neurons are highly activated for a given input?
INPUT OUTPUTMODEL
Cat
Dog ...
tags: #mycat, #cute
date: 10/1/2017
location: 33.7, 88.4
Practical Challenges in Industry
5
DIVERSE INPUT TYPESCOMPLEX MOD...
Understanding ActiVis Users’ Needs
Participatory design sessions over 11 months, with
15+ Facebook researchers, engineers,...
Complex Model Architectures
UNDERSTANDING ACTIVIS USERS’ NEEDS (1/3)
7
Numerous deep & wide models are used.
input
output
...
Huge Datasets with Diverse Features
UNDERSTANDING ACTIVIS USERS’ NEEDS (2/3)
1 billion+
instances
8
1,000+ multi-type feat...
Two Key Analytics Patterns
UNDERSTANDING ACTIVIS USERS’ NEEDS (3/3)
How model responds to
individual instances?
(This inst...
ActiVis Design Goals (Recap)
10
Two analytics patterns
Complex models
Huge datasets
1.
2.
3. Unified analysis for instance...
ActiVis
Visualizing activation of industry-scale deep neural nets
Deployed on Facebook ML Platform
11
12
demo:
“Where is Phoenix located?” → location
“What is the diameter of a golf ball?” → numeric
Exploring text classifica...
Demo
13
• Unified analysis for instances & subsets
• Model architecture to activation details
• Scaling to industry-scale data & m...
User-guided Instance Sampling
15
SCALING TO LARGE DATA & MODELS (1/2)
e.g., “What does VAST mean?”
should be in ABBR class...
Selective Precomputation for Important Nodes
16
SCALING TO LARGE DATA & MODELS (2/2)
Often only a few nodes are helpful an...
Deployed on FBLearner
17
Model developers add 3 API calls to enable ActiVis.
Facebook ML platform, used by 25% of their en...
Case Studies
3 Facebook participants.
All work with text classification models.
Each session 60 minutes long.
18
Spot1. -checking models with “test cases”
“Where is … located?” → location
Graph architecture view as entry point2.
Debugg...
• Discover interesting subsets interactively
• Support input-dependent models (e.g., RNN)
• Provide direct guidance for pe...
ActiVis
Visual Exploration of Industry-Scale
Deep Neural Network Models
✓ Deployed on Facebook’s ML platform
✓ Support sub...
Upcoming SlideShare
Loading in …5
×

ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models

1,550 views

Published on

Presentation slides for the following paper presented at IEEE VIS 2017 (http://ieeevis.org).
Minsuk Kahng, Pierre Y. Andrews, Aditya Kalro, and Duen Horng (Polo) Chau. ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models. IEEE Transactions on Visualization and Computer Graphics, Vol. 24, No. 1 (VAST 2017).

Published in: Software
  • Be the first to comment

ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models

  1. 1. ActiVis Visual Exploration of Industry-Scale Deep Neural Network Models Minsuk Kahng Pierre Andrews Aditya Kalro Polo Chau Georgia Tech Georgia TechFacebook Facebook *
  2. 2. Deep Learning is Powerful 2Image credit:http://www.nvidia.com/object/drive-px.html, https://venturebeat.com/2016/04/14/, https://finance.yahoo.com/news/
  3. 3. Understanding Deep Learning is Challenging 3 Cat Dog Cat Dog INPUT OUTPUTMODEL Image credit: https://www.kaggle.com/c/dogs-vs-cats/ Incorrect
  4. 4. Cat Dog Visualizing Instance Activation 4 Which neurons are highly activated for a given input? INPUT OUTPUTMODEL Cat Dog 78% 22% [Yosinski et al., 2015; Harley, 2015; Karpathy, 2016; Liu et al., 2017]
  5. 5. tags: #mycat, #cute date: 10/1/2017 location: 33.7, 88.4 Practical Challenges in Industry 5 DIVERSE INPUT TYPESCOMPLEX MODELS LARGE DATASETS image, text, numerical, categorical, … many nodes in graph-structure 1 billion+ instances 1. 2. 3. Develop ActiVis for Facebook-scale models and dataGOAL: Enjoying nice weather with kiki❤️
  6. 6. Understanding ActiVis Users’ Needs Participatory design sessions over 11 months, with 15+ Facebook researchers, engineers, & data scientists 6
  7. 7. Complex Model Architectures UNDERSTANDING ACTIVIS USERS’ NEEDS (1/3) 7 Numerous deep & wide models are used. input output Separate architecture from activation detailsGOAL:
  8. 8. Huge Datasets with Diverse Features UNDERSTANDING ACTIVIS USERS’ NEEDS (2/3) 1 billion+ instances 8 1,000+ multi-type features (e.g., image, text, numerical, categorical) tags: #mycat, #cute date: 10/1/2017 location: 33.7, 88.4 Enjoying nice weather with kiki❤️ Use multiple approaches for scalability GOAL: Make use of diverse features
  9. 9. Two Key Analytics Patterns UNDERSTANDING ACTIVIS USERS’ NEEDS (3/3) How model responds to individual instances? (This instance highly activates neurons #2, 5, 11.) How model behaves at higher-level categorization (e.g., by topic)? (A subset of instances about “sports” highly activates neurons #3, 7, 11.) Useful for debugging Useful for large datasets SUBSET-LEVELINSTANCE-LEVEL 9 [Kahng et al., 2016; Krause et al., 2016] Complementary [Kulesza et al., 2015; Amershi et al., 2015] Support both analyticsGOAL:
  10. 10. ActiVis Design Goals (Recap) 10 Two analytics patterns Complex models Huge datasets 1. 2. 3. Unified analysis for instances & subsets Model overview as entry point Multiple approaches for scalability
  11. 11. ActiVis Visualizing activation of industry-scale deep neural nets Deployed on Facebook ML Platform 11
  12. 12. 12 demo: “Where is Phoenix located?” → location “What is the diameter of a golf ball?” → numeric Exploring text classification results ActiVis Demo
  13. 13. Demo 13
  14. 14. • Unified analysis for instances & subsets • Model architecture to activation details • Scaling to industry-scale data & models 14 ActiVis Key Ideas (Demo Recap)
  15. 15. User-guided Instance Sampling 15 SCALING TO LARGE DATA & MODELS (1/2) e.g., “What does VAST mean?” should be in ABBR class. Users either want representative samples or maintain “test cases”.
  16. 16. Selective Precomputation for Important Nodes 16 SCALING TO LARGE DATA & MODELS (2/2) Often only a few nodes are helpful and model developers know them. precomputed
  17. 17. Deployed on FBLearner 17 Model developers add 3 API calls to enable ActiVis. Facebook ML platform, used by 25% of their engineers Click to launch ActiVis
  18. 18. Case Studies 3 Facebook participants. All work with text classification models. Each session 60 minutes long. 18
  19. 19. Spot1. -checking models with “test cases” “Where is … located?” → location Graph architecture view as entry point2. Debugging hints from activation patterns3. 19 Key Observations from Case Studies
  20. 20. • Discover interesting subsets interactively • Support input-dependent models (e.g., RNN) • Provide direct guidance for performance improvement 20 Future Research Directions
  21. 21. ActiVis Visual Exploration of Industry-Scale Deep Neural Network Models ✓ Deployed on Facebook’s ML platform ✓ Support subset-level analysis ✓ Model architecture to activation details We thank Facebook Applied Machine Learning Group, especially Yangqing Jia, Andrew Tulloch, Liang Xiong, and Zhao Tan and NSF Graduate Research Fellowship Program. Pierre Andrews Aditya Kalro Polo Chau Georgia Tech Facebook Facebook Minsuk Kahng Georgia Tech PhD student http://minsuk.com

×