SlideShare a Scribd company logo
1 of 13
Developing Qualitative Metrics for Visual Analytic Environments Jean Scholtz Pacific Northwest National Laboratory BELIV 2010
Why Are Qualitative Metrics Needed? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Qualitative Assessment and the VAST Challenge ,[object Object],[object Object],[object Object],[object Object]
Evaluating the Reviews ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What materials should be provided to evaluators to ensure that they can adequately assess the visual analytic systems?  ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What aspects of visual analytic systems are important to reviewers? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Comments on  Analytic Process ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Comments on Visualizations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Comments on Interactions ,[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Is there an advantage to selecting evaluators from different domains of expertise? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Why Visualization Researchers Gave Lower Ratings ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Overall Conclusions ,[object Object],[object Object],[object Object]

More Related Content

More from BELIV Workshop

A Descriptive Model of Visual Scanning.
A Descriptive Model of Visual Scanning.A Descriptive Model of Visual Scanning.
A Descriptive Model of Visual Scanning.BELIV Workshop
 
Generating a synthetic video dataset
Generating a synthetic video datasetGenerating a synthetic video dataset
Generating a synthetic video datasetBELIV Workshop
 
Beyond system logging: human logging for evaluating information visualization.
Beyond system logging: human logging for evaluating information visualization.Beyond system logging: human logging for evaluating information visualization.
Beyond system logging: human logging for evaluating information visualization.BELIV Workshop
 
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.BELIV Workshop
 
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.BELIV Workshop
 
Learning-Based Evaluation of Visual Analytic Systems.
Learning-Based Evaluation of Visual Analytic Systems.Learning-Based Evaluation of Visual Analytic Systems.
Learning-Based Evaluation of Visual Analytic Systems.BELIV Workshop
 
Visualization Evaluation of the Masses, by the Masses, and for the Masses.
Visualization Evaluation of the Masses, by the Masses, and for the Masses.Visualization Evaluation of the Masses, by the Masses, and for the Masses.
Visualization Evaluation of the Masses, by the Masses, and for the Masses.BELIV Workshop
 
Evaluating Information Visualization in Large Companies: Challenges, Experien...
Evaluating Information Visualization in Large Companies: Challenges, Experien...Evaluating Information Visualization in Large Companies: Challenges, Experien...
Evaluating Information Visualization in Large Companies: Challenges, Experien...BELIV Workshop
 
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.BELIV Workshop
 
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...BELIV Workshop
 
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz EvaluationsBELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz EvaluationsBELIV Workshop
 

More from BELIV Workshop (11)

A Descriptive Model of Visual Scanning.
A Descriptive Model of Visual Scanning.A Descriptive Model of Visual Scanning.
A Descriptive Model of Visual Scanning.
 
Generating a synthetic video dataset
Generating a synthetic video datasetGenerating a synthetic video dataset
Generating a synthetic video dataset
 
Beyond system logging: human logging for evaluating information visualization.
Beyond system logging: human logging for evaluating information visualization.Beyond system logging: human logging for evaluating information visualization.
Beyond system logging: human logging for evaluating information visualization.
 
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
Scanning Between Graph Visualizations: An Eye Tracking Evaluation.
 
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
Focus Groups for Functional InfoVis Prototype Evaluation: A Case Study.
 
Learning-Based Evaluation of Visual Analytic Systems.
Learning-Based Evaluation of Visual Analytic Systems.Learning-Based Evaluation of Visual Analytic Systems.
Learning-Based Evaluation of Visual Analytic Systems.
 
Visualization Evaluation of the Masses, by the Masses, and for the Masses.
Visualization Evaluation of the Masses, by the Masses, and for the Masses.Visualization Evaluation of the Masses, by the Masses, and for the Masses.
Visualization Evaluation of the Masses, by the Masses, and for the Masses.
 
Evaluating Information Visualization in Large Companies: Challenges, Experien...
Evaluating Information Visualization in Large Companies: Challenges, Experien...Evaluating Information Visualization in Large Companies: Challenges, Experien...
Evaluating Information Visualization in Large Companies: Challenges, Experien...
 
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.
 
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
Comparative Evaluation of Two Interface Tools in Performing Visual Analytics ...
 
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz EvaluationsBELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
BELIV'10 Keynote: Conceptual and Practical Challenges in InfoViz Evaluations
 

Recently uploaded

A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxBkGupta21
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????blackmambaettijean
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 

Recently uploaded (20)

A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptx
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 

Developing Qualitative Metrics for Visual Analytic Environments

  • 1. Developing Qualitative Metrics for Visual Analytic Environments Jean Scholtz Pacific Northwest National Laboratory BELIV 2010
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.