SlideShare a Scribd company logo
1 of 13
Developing Qualitative Metrics for Visual Analytic Environments Jean Scholtz Pacific Northwest National Laboratory BELIV 2010
Why Are Qualitative Metrics Needed? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Qualitative Assessment and the VAST Challenge ,[object Object],[object Object],[object Object],[object Object]
Evaluating the Reviews ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What materials should be provided to evaluators to ensure that they can adequately assess the visual analytic systems?  ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What aspects of visual analytic systems are important to reviewers? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Comments on  Analytic Process ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Comments on Visualizations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Comments on Interactions ,[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Is there an advantage to selecting evaluators from different domains of expertise? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Why Visualization Researchers Gave Lower Ratings ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Overall Conclusions ,[object Object],[object Object],[object Object]

More Related Content

More from BELIV Workshop

A Descriptive Model of Visual Scanning.
A Descriptive Model of Visual Scanning.A Descriptive Model of Visual Scanning.
A Descriptive Model of Visual Scanning.BELIV Workshop
 
Generating a synthetic video dataset
Generating a synthetic video datasetGenerating a synthetic video dataset
Generating a synthetic video datasetBELIV Workshop
 
Beyond system logging: human logging for evaluating information visualization.
Beyond system logging: human logging for evaluating information visualization.Beyond system logging: human logging for evaluating information visualization.
Beyond system logging: human logging for evaluating information visualization.BELIV Workshop
 
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.BELIV Workshop
 
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.BELIV Workshop
 
Learning-Based Evaluation of Visual Analytic Systems.
Learning-Based Evaluation of Visual Analytic Systems.Learning-Based Evaluation of Visual Analytic Systems.
Learning-Based Evaluation of Visual Analytic Systems.BELIV Workshop
 
Visualization Evaluation of the Masses, by the Masses, and for the Masses.
Visualization Evaluation of the Masses, by the Masses, and for the Masses.Visualization Evaluation of the Masses, by the Masses, and for the Masses.
Visualization Evaluation of the Masses, by the Masses, and for the Masses.BELIV Workshop
 
Evaluating Information Visualization in Large Companies: Challenges, Experien...
Evaluating Information Visualization in Large Companies: Challenges, Experien...Evaluating Information Visualization in Large Companies: Challenges, Experien...
Evaluating Information Visualization in Large Companies: Challenges, Experien...BELIV Workshop
 
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.BELIV Workshop
 
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...BELIV Workshop
 
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz EvaluationsBELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz EvaluationsBELIV Workshop
 

More from BELIV Workshop (11)

A Descriptive Model of Visual Scanning.
A Descriptive Model of Visual Scanning.A Descriptive Model of Visual Scanning.
A Descriptive Model of Visual Scanning.
 
Generating a synthetic video dataset
Generating a synthetic video datasetGenerating a synthetic video dataset
Generating a synthetic video dataset
 
Beyond system logging: human logging for evaluating information visualization.
Beyond system logging: human logging for evaluating information visualization.Beyond system logging: human logging for evaluating information visualization.
Beyond system logging: human logging for evaluating information visualization.
 
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
 
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
 
Learning-Based Evaluation of Visual Analytic Systems.
Learning-Based Evaluation of Visual Analytic Systems.Learning-Based Evaluation of Visual Analytic Systems.
Learning-Based Evaluation of Visual Analytic Systems.
 
Visualization Evaluation of the Masses, by the Masses, and for the Masses.
Visualization Evaluation of the Masses, by the Masses, and for the Masses.Visualization Evaluation of the Masses, by the Masses, and for the Masses.
Visualization Evaluation of the Masses, by the Masses, and for the Masses.
 
Evaluating Information Visualization in Large Companies: Challenges, Experien...
Evaluating Information Visualization in Large Companies: Challenges, Experien...Evaluating Information Visualization in Large Companies: Challenges, Experien...
Evaluating Information Visualization in Large Companies: Challenges, Experien...
 
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
 
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
 
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz EvaluationsBELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
 

Recently uploaded

The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Key Features Of Token Development (1).pptx
Key  Features Of Token  Development (1).pptxKey  Features Of Token  Development (1).pptx
Key Features Of Token Development (1).pptxLBM Solutions
 
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAzure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAndikSusilo4
 
WhatsApp 9892124323 âś“Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 âś“Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 âś“Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 âś“Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 

Recently uploaded (20)

The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptx
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Key Features Of Token Development (1).pptx
Key  Features Of Token  Development (1).pptxKey  Features Of Token  Development (1).pptx
Key Features Of Token Development (1).pptx
 
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAzure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & Application
 
WhatsApp 9892124323 âś“Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 âś“Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 âś“Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 âś“Call Girls In Kalyan ( Mumbai ) secure service
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 

Developing Qualitative Metrics for Visual Analytic Environments

  • 1. Developing Qualitative Metrics for Visual Analytic Environments Jean Scholtz Pacific Northwest National Laboratory BELIV 2010
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.