This document summarizes a presentation given by Katrien Verbert on explainable artificial intelligence and interactive explanation methods. It discusses Verbert's research group at KU Leuven which focuses on areas like recommender systems, visualization, and intelligent user interfaces. The presentation provides an overview of explainable AI, discussing objectives like explaining model outcomes to increase trust and allowing user interaction with explanations. It describes various recommendation techniques and presents examples of explainable recommendation systems. The presentation discusses how personal user characteristics can impact the effects of explanations and outlines related user studies. Finally, it summarizes several of Verbert's application areas for explainable AI like education, analytics, agriculture, and healthcare, touching on methodologies and results.
Keynote at Chilean Week of Computer Science. I present a brief overview of algorithms for Recommender and then I present my work Tag-based Recommendation, Implicit Feedback and Visual Interactive Interfaces.
User Control in AIED (Artificial Intelligence in Education)Peter Brusilovsky
Slides of my intro to "Meet the Expert" session at AIED 2021. This is a subset of slides of a longer presentation on user control in AI extended with many specific examples from AIED area.
Keynote at Chilean Week of Computer Science. I present a brief overview of algorithms for Recommender and then I present my work Tag-based Recommendation, Implicit Feedback and Visual Interactive Interfaces.
User Control in AIED (Artificial Intelligence in Education)Peter Brusilovsky
Slides of my intro to "Meet the Expert" session at AIED 2021. This is a subset of slides of a longer presentation on user control in AI extended with many specific examples from AIED area.
Iui2015: Personalized Search: Reconsidering the Value of Open User ModelsPeter Brusilovsky
IUI 2015 talk slides: Ahn, J., Brusilovsky, P., and Han, S. (2015) Personalized Search: Reconsidering the Value of Open User Models. In: Proceedings of Proceedings of the 20th International Conference on Intelligent User Interfaces, Atlanta, Georgia, USA, ACM, pp. 202-212
Human-Centered AI in AI-ED - Keynote at AAAI 2022 AI for Education workshopPeter Brusilovsky
Abstract: In recent years, the use of Artificial Intelligence (AI) technologies expanded to many areas directly affecting the lives of millions. AI-based approaches advise human decision-makers who should be released on bail, whether it is a good time to discharge a patient from a hospital and whether a specific student is at risk to fail a course. Such extensive use in AI in decision making came with a range of protentional problems that have been extensively studied over the last few years. Recognition of these problems motivated a rapid rise of research on “human-centered AI”, which attempted to address and minimize the negative effects of using AI technologies. The majority of work on human-centered AI focus on various types of Human-AI collaboration through such technologies as transparency, explainability, and user control. In my talk, I will review how the ideas of Human-AI collaboration, transparency, explainability, and user control have been used in educational applications of AI in the past and will discuss now new ideas in this research area developed outside of AI-Ed could be creatively applied in educational context.
Best Practices in Recommender System ChallengesAlan Said
Recommender System Challenges such as the Netflix Prize, KDD Cup, etc. have contributed vastly to the development and adoptability of recommender systems. Each year a number of challenges or contests are organized covering different aspects of recommendation. In this tutorial and panel, we present some of the factors involved in successfully organizing a challenge, whether for reasons purely related to research, industrial challenges, or to widen the scope of recommender systems applications.
Analysis of Multiple Pilots for ICT-supported Lifelong Competence Development, Davinia Hernández-Leo, davinia.hernandez@upf.edu, TENCompetence Winter School 2009, 1-6 February Innsbruck, Austria
Two Brains are Better than One: User Control in Adaptive Information AccessPeter Brusilovsky
In recent years, the use of Artificial Intelligence (AI) technologies expanded to many areas where they directly affect the lives of many people. AI-based approaches advise human decision-makers who should be released on bail, whether it is a good time to discharge a patient from a hospital and whether a specific student is at risk to fail a course. Such an extensive use in AI in decision making came with a range of protentional problems that have been extensively studied over the last few years. Recognition of these problems motivated a rapid rise of research on “human-centered AI”, which attempted to address and minimize the negative effects of using AI technologies. Among the ideas of human-centered AI is user control - engaging users in affecting AI decision making to prevent possible errors and biases. In my talk, I will focus on the application of user control in one popular area of AI application, adaptive information access. Adaptive information access systems such as personalized search and recommender systems attempt to model their users to help them in finding the most relevant information. Yet, user modeling and personalization mechanisms might not always work as expected resulting in errors, biases, and suboptimal behavior. Combining the decision power or AI with the ability of the user to guide and control it brings together the strong sides of artificial and human intelligence and could lead to better results. In my talk, I review several projects focused on user control in adaptive information access systems and discuss the benefits and challenges of this approach.
Data-Driven Education 2020: Using Big Educational Data to Improve Teaching an...Peter Brusilovsky
Modern educational settings from regular classrooms to MOOCs produce a a rapidly increasing volume of data that captures individual learning progress of millions of students at different level of granularity. This presence of this data opens a unique opportunity to re-engineer traditional education and build and develop a range of efficient data-driven approaches to support teaching and learning. In my talk, I will present several ways to use big educational data explored in our lab. The focus will be on open social learning modeling and identifying individual differences through sequential pattern mining, but several other approaches will be mentioned. Open social learning modeling and sequential pattern mining provides two considerably different examples on using educational data. One offers an immediate use of class interaction history to develop more engaging content access while another shows how big data could be used to uncover important individual differences that could be used to optimize the process for individual leaners.
Stereotype Modeling for Problem-Solving Performance Predictions in MOOCs and ...Peter Brusilovsky
Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, UMAP2017, pp 76-84
Stereotypes are frequently used in real life to classify students according to their performance in class. In literature, we can find many references to weaker students, fast learners, struggling students, etc. Given the lack of detailed data about students, these or other kinds of stereotypes could be potentially used for user modeling and personalization in the educational context. Recent research in MOOC context demonstrated that data-driven learner stereotypes could work well for detecting and preventing student dropouts. In this paper, we are exploring the application of stereotype-based modeling to a more challenging task -- predicting student problem-solving and learning in two programming courses and two MOOCs. We explore traditional stereotypes based on readily available factors like gender or education level as well as some advanced data-driven approaches to group students based on their problem-solving behavior. Each of the approaches to form student stereotype cohorts is validated by comparing models of student learning: do students in different groups learn differently? In the search for the stereotypes that could be used for adaptation, the paper examines ten approaches. We compare the performance of these approaches and draw conclusions for future research.
Analyzing sentiment system to specify polarity by lexicon-basedjournalBEEI
Currently, sentiment analysis into positive or negative getting more attention from the researchers. With the rapid development of the internet and social media have made people express their views and opinion publicly. Analyzing the sentiment in people views and opinion impact many fields such as services and productions that companies offer. Movie reviewer needs many processing to be prepared to detect emotion, classify them and achieve high accuracy. The difficulties arise due of the structure and grammar of the language and manage the dictionary. We present a system that assigns scores indicating positive or negative opinion to each distinct entity in the text corpus. Propose an innovative formula to compute the polarity score for each word occurring in the text and find it in positive dictionary or negative dictionary we have to remove it from text. After classification, the words are stored in a list that will be used to calculate the accuracy. The results reveal that the system achieved the best results in accuracy of 76.585%.
The Innovation Engine for Team Building – The EU Aristotele Approach From Ope...ARISTOTELE
ARISTOTELE approach has been presented at the Innovation Adoption Forum for Industry and Public Sector within the 6th IEEE International Conference on Digital Ecosystem Technologies (IEEE DEST - CEE 2012). The presentation about ARISTOTELE has been held by Paolo Ceravolo and Ernesto Damiani (University of Milan) during the keynote "The Innovation Engine for Team Building – The EU Aristotele Approach". Learn more on http://www.aristotele-ip.eu/
USING NLP APPROACH FOR ANALYZING CUSTOMER REVIEWScsandit
The Web considers one of the main sources of customer opinions and reviews which they are represented in two formats; structured data (numeric ratings) and unstructured data (textual comments). Millions of textual comments about goods and services are posted on the web by customers and every day thousands are added, make it a big challenge to read and understand them to make them a useful structured data for customers and decision makers. Sentiment
analysis or Opinion mining is a popular technique for summarizing and analyzing those opinions and reviews. In this paper, we use natural language processing techniques to generate some rules to help us understand customer opinions and reviews (textual comments) written in the Arabic language for the purpose of understanding each one of them and then convert them to a structured data. We use adjectives as a key point to highlight important information in the text then we work around them to tag attributes that describe the subject of the reviews, and we associate them with their values (adjectives).
Iui2015: Personalized Search: Reconsidering the Value of Open User ModelsPeter Brusilovsky
IUI 2015 talk slides: Ahn, J., Brusilovsky, P., and Han, S. (2015) Personalized Search: Reconsidering the Value of Open User Models. In: Proceedings of Proceedings of the 20th International Conference on Intelligent User Interfaces, Atlanta, Georgia, USA, ACM, pp. 202-212
Human-Centered AI in AI-ED - Keynote at AAAI 2022 AI for Education workshopPeter Brusilovsky
Abstract: In recent years, the use of Artificial Intelligence (AI) technologies expanded to many areas directly affecting the lives of millions. AI-based approaches advise human decision-makers who should be released on bail, whether it is a good time to discharge a patient from a hospital and whether a specific student is at risk to fail a course. Such extensive use in AI in decision making came with a range of protentional problems that have been extensively studied over the last few years. Recognition of these problems motivated a rapid rise of research on “human-centered AI”, which attempted to address and minimize the negative effects of using AI technologies. The majority of work on human-centered AI focus on various types of Human-AI collaboration through such technologies as transparency, explainability, and user control. In my talk, I will review how the ideas of Human-AI collaboration, transparency, explainability, and user control have been used in educational applications of AI in the past and will discuss now new ideas in this research area developed outside of AI-Ed could be creatively applied in educational context.
Best Practices in Recommender System ChallengesAlan Said
Recommender System Challenges such as the Netflix Prize, KDD Cup, etc. have contributed vastly to the development and adoptability of recommender systems. Each year a number of challenges or contests are organized covering different aspects of recommendation. In this tutorial and panel, we present some of the factors involved in successfully organizing a challenge, whether for reasons purely related to research, industrial challenges, or to widen the scope of recommender systems applications.
Analysis of Multiple Pilots for ICT-supported Lifelong Competence Development, Davinia Hernández-Leo, davinia.hernandez@upf.edu, TENCompetence Winter School 2009, 1-6 February Innsbruck, Austria
Two Brains are Better than One: User Control in Adaptive Information AccessPeter Brusilovsky
In recent years, the use of Artificial Intelligence (AI) technologies expanded to many areas where they directly affect the lives of many people. AI-based approaches advise human decision-makers who should be released on bail, whether it is a good time to discharge a patient from a hospital and whether a specific student is at risk to fail a course. Such an extensive use in AI in decision making came with a range of protentional problems that have been extensively studied over the last few years. Recognition of these problems motivated a rapid rise of research on “human-centered AI”, which attempted to address and minimize the negative effects of using AI technologies. Among the ideas of human-centered AI is user control - engaging users in affecting AI decision making to prevent possible errors and biases. In my talk, I will focus on the application of user control in one popular area of AI application, adaptive information access. Adaptive information access systems such as personalized search and recommender systems attempt to model their users to help them in finding the most relevant information. Yet, user modeling and personalization mechanisms might not always work as expected resulting in errors, biases, and suboptimal behavior. Combining the decision power or AI with the ability of the user to guide and control it brings together the strong sides of artificial and human intelligence and could lead to better results. In my talk, I review several projects focused on user control in adaptive information access systems and discuss the benefits and challenges of this approach.
Data-Driven Education 2020: Using Big Educational Data to Improve Teaching an...Peter Brusilovsky
Modern educational settings from regular classrooms to MOOCs produce a a rapidly increasing volume of data that captures individual learning progress of millions of students at different level of granularity. This presence of this data opens a unique opportunity to re-engineer traditional education and build and develop a range of efficient data-driven approaches to support teaching and learning. In my talk, I will present several ways to use big educational data explored in our lab. The focus will be on open social learning modeling and identifying individual differences through sequential pattern mining, but several other approaches will be mentioned. Open social learning modeling and sequential pattern mining provides two considerably different examples on using educational data. One offers an immediate use of class interaction history to develop more engaging content access while another shows how big data could be used to uncover important individual differences that could be used to optimize the process for individual leaners.
Stereotype Modeling for Problem-Solving Performance Predictions in MOOCs and ...Peter Brusilovsky
Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, UMAP2017, pp 76-84
Stereotypes are frequently used in real life to classify students according to their performance in class. In literature, we can find many references to weaker students, fast learners, struggling students, etc. Given the lack of detailed data about students, these or other kinds of stereotypes could be potentially used for user modeling and personalization in the educational context. Recent research in MOOC context demonstrated that data-driven learner stereotypes could work well for detecting and preventing student dropouts. In this paper, we are exploring the application of stereotype-based modeling to a more challenging task -- predicting student problem-solving and learning in two programming courses and two MOOCs. We explore traditional stereotypes based on readily available factors like gender or education level as well as some advanced data-driven approaches to group students based on their problem-solving behavior. Each of the approaches to form student stereotype cohorts is validated by comparing models of student learning: do students in different groups learn differently? In the search for the stereotypes that could be used for adaptation, the paper examines ten approaches. We compare the performance of these approaches and draw conclusions for future research.
Analyzing sentiment system to specify polarity by lexicon-basedjournalBEEI
Currently, sentiment analysis into positive or negative getting more attention from the researchers. With the rapid development of the internet and social media have made people express their views and opinion publicly. Analyzing the sentiment in people views and opinion impact many fields such as services and productions that companies offer. Movie reviewer needs many processing to be prepared to detect emotion, classify them and achieve high accuracy. The difficulties arise due of the structure and grammar of the language and manage the dictionary. We present a system that assigns scores indicating positive or negative opinion to each distinct entity in the text corpus. Propose an innovative formula to compute the polarity score for each word occurring in the text and find it in positive dictionary or negative dictionary we have to remove it from text. After classification, the words are stored in a list that will be used to calculate the accuracy. The results reveal that the system achieved the best results in accuracy of 76.585%.
The Innovation Engine for Team Building – The EU Aristotele Approach From Ope...ARISTOTELE
ARISTOTELE approach has been presented at the Innovation Adoption Forum for Industry and Public Sector within the 6th IEEE International Conference on Digital Ecosystem Technologies (IEEE DEST - CEE 2012). The presentation about ARISTOTELE has been held by Paolo Ceravolo and Ernesto Damiani (University of Milan) during the keynote "The Innovation Engine for Team Building – The EU Aristotele Approach". Learn more on http://www.aristotele-ip.eu/
USING NLP APPROACH FOR ANALYZING CUSTOMER REVIEWScsandit
The Web considers one of the main sources of customer opinions and reviews which they are represented in two formats; structured data (numeric ratings) and unstructured data (textual comments). Millions of textual comments about goods and services are posted on the web by customers and every day thousands are added, make it a big challenge to read and understand them to make them a useful structured data for customers and decision makers. Sentiment
analysis or Opinion mining is a popular technique for summarizing and analyzing those opinions and reviews. In this paper, we use natural language processing techniques to generate some rules to help us understand customer opinions and reviews (textual comments) written in the Arabic language for the purpose of understanding each one of them and then convert them to a structured data. We use adjectives as a key point to highlight important information in the text then we work around them to tag attributes that describe the subject of the reviews, and we associate them with their values (adjectives).
Scalable Exploration of Relevance Prospects to Support Decision MakingKatrien Verbert
Presented at IntRS 2016 - Interfaces and Human Decision Making for Recommender Systems, workshop at RecSys 2016
Citation: Verbert, K., Seipp, K., He, C., Parra, D., Wongchokprasitti, C., & Brusilovsky, P. (2016). Scalable Exploration of Relevance Prospects to Support Decision Making. Proceedings of the Joint Workshop on Interfaces and Human Decision Making for Recommender Systems co-located with ACM Conference on Recommender Systems (RecSys 2016), Boston, MA, USA, September 16, 2016.
Recent Research and Developments on Recommender Systems in TELHendrik Drachsler
Presentation given at the Learning Network seminar series at CELSTEC. Special guest was Wolfgang Reinhardt who provided his view on data science in relation to awareness improvement of knowledge workers.
Agents vs Users: Visual Recommendation of Research Talks with Multiple Dimens...Katrien Verbert
Published in ACM TiiS: Verbert, K., Parra, D., & Brusilovsky, P. (2016). Agents Vs. Users: Visual Recommendation of Research Talks with Multiple Dimension of Relevance. ACM Transactions on Interactive Intelligent Systems (TiiS), 6(2), 11.
Presented at IUI 2017
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
Cosmetic shop management system project report.pdfKamal Acharya
Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry.
Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks.
Data file handling has been effectively used in the program.
The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.
COLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdfKamal Acharya
The College Bus Management system is completely developed by Visual Basic .NET Version. The application is connect with most secured database language MS SQL Server. The application is develop by using best combination of front-end and back-end languages. The application is totally design like flat user interface. This flat user interface is more attractive user interface in 2017. The application is gives more important to the system functionality. The application is to manage the student’s details, driver’s details, bus details, bus route details, bus fees details and more. The application has only one unit for admin. The admin can manage the entire application. The admin can login into the application by using username and password of the admin. The application is develop for big and small colleges. It is more user friendly for non-computer person. Even they can easily learn how to manage the application within hours. The application is more secure by the admin. The system will give an effective output for the VB.Net and SQL Server given as input to the system. The compiled java program given as input to the system, after scanning the program will generate different reports. The application generates the report for users. The admin can view and download the report of the data. The application deliver the excel format reports. Because, excel formatted reports is very easy to understand the income and expense of the college bus. This application is mainly develop for windows operating system users. In 2017, 73% of people enterprises are using windows operating system. So the application will easily install for all the windows operating system users. The application-developed size is very low. The application consumes very low space in disk. Therefore, the user can allocate very minimum local disk space for this application.
Explore the innovative world of trenchless pipe repair with our comprehensive guide, "The Benefits and Techniques of Trenchless Pipe Repair." This document delves into the modern methods of repairing underground pipes without the need for extensive excavation, highlighting the numerous advantages and the latest techniques used in the industry.
Learn about the cost savings, reduced environmental impact, and minimal disruption associated with trenchless technology. Discover detailed explanations of popular techniques such as pipe bursting, cured-in-place pipe (CIPP) lining, and directional drilling. Understand how these methods can be applied to various types of infrastructure, from residential plumbing to large-scale municipal systems.
Ideal for homeowners, contractors, engineers, and anyone interested in modern plumbing solutions, this guide provides valuable insights into why trenchless pipe repair is becoming the preferred choice for pipe rehabilitation. Stay informed about the latest advancements and best practices in the field.
TECHNICAL TRAINING MANUAL GENERAL FAMILIARIZATION COURSEDuvanRamosGarzon1
AIRCRAFT GENERAL
The Single Aisle is the most advanced family aircraft in service today, with fly-by-wire flight controls.
The A318, A319, A320 and A321 are twin-engine subsonic medium range aircraft.
The family offers a choice of engines
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdffxintegritypublishin
Advancements in technology unveil a myriad of electrical and electronic breakthroughs geared towards efficiently harnessing limited resources to meet human energy demands. The optimization of hybrid solar PV panels and pumped hydro energy supply systems plays a pivotal role in utilizing natural resources effectively. This initiative not only benefits humanity but also fosters environmental sustainability. The study investigated the design optimization of these hybrid systems, focusing on understanding solar radiation patterns, identifying geographical influences on solar radiation, formulating a mathematical model for system optimization, and determining the optimal configuration of PV panels and pumped hydro storage. Through a comparative analysis approach and eight weeks of data collection, the study addressed key research questions related to solar radiation patterns and optimal system design. The findings highlighted regions with heightened solar radiation levels, showcasing substantial potential for power generation and emphasizing the system's efficiency. Optimizing system design significantly boosted power generation, promoted renewable energy utilization, and enhanced energy storage capacity. The study underscored the benefits of optimizing hybrid solar PV panels and pumped hydro energy supply systems for sustainable energy usage. Optimizing the design of solar PV panels and pumped hydro energy supply systems as examined across diverse climatic conditions in a developing country, not only enhances power generation but also improves the integration of renewable energy sources and boosts energy storage capacities, particularly beneficial for less economically prosperous regions. Additionally, the study provides valuable insights for advancing energy research in economically viable areas. Recommendations included conducting site-specific assessments, utilizing advanced modeling tools, implementing regular maintenance protocols, and enhancing communication among system components.
Saudi Arabia stands as a titan in the global energy landscape, renowned for its abundant oil and gas resources. It's the largest exporter of petroleum and holds some of the world's most significant reserves. Let's delve into the top 10 oil and gas projects shaping Saudi Arabia's energy future in 2024.
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxR&R Consult
CFD analysis is incredibly effective at solving mysteries and improving the performance of complex systems!
Here's a great example: At a large natural gas-fired power plant, where they use waste heat to generate steam and energy, they were puzzled that their boiler wasn't producing as much steam as expected.
R&R and Tetra Engineering Group Inc. were asked to solve the issue with reduced steam production.
An inspection had shown that a significant amount of hot flue gas was bypassing the boiler tubes, where the heat was supposed to be transferred.
R&R Consult conducted a CFD analysis, which revealed that 6.3% of the flue gas was bypassing the boiler tubes without transferring heat. The analysis also showed that the flue gas was instead being directed along the sides of the boiler and between the modules that were supposed to capture the heat. This was the cause of the reduced performance.
Based on our results, Tetra Engineering installed covering plates to reduce the bypass flow. This improved the boiler's performance and increased electricity production.
It is always satisfying when we can help solve complex challenges like this. Do your systems also need a check-up or optimization? Give us a call!
Work done in cooperation with James Malloy and David Moelling from Tetra Engineering.
More examples of our work https://www.r-r-consult.dk/en/cases-en/
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)MdTanvirMahtab2
This presentation is about the working procedure of Shahjalal Fertilizer Company Limited (SFCL). A Govt. owned Company of Bangladesh Chemical Industries Corporation under Ministry of Industries.
Towards the next generation of interactive and adaptive explanation methods
1. Towards the next generation of interactive
and adaptive explanation methods
IWM-Lecture series - 7 Dec 2021
Katrien Verbert
Augment/HCI - KU Leuven
@katrien_v
2. Human-Computer Interaction group
Explainable AI - recommender systems – visualization – intelligent user interfaces
Learning analytics &
human resources
Media
consumption
Precision agriculture
Healthcare
Augment prof. Katrien Verbert
ARIA prof. Adalberto Simeone
Computer
Graphics
prof. Phil Dutré
Language
Intelligence &
Information
Retrieval
prof. Sien Moens
3. Augment/HCI team
Robin De Croon
Postdoc researcher
Katrien Verbert
Associate Professor
Francisco Gutiérrez
Postdoc researcher
Tom Broos
PhD researcher
Nyi Nyi Htun
Postdoc researcher
Houda Lamqaddam
PhD researcher
Oscar Alvarado
Postdoc researcher
http://augment.cs.kuleuven.be/
Diego Rojo Carcia
PhD researcher
Maxwell Szymanski
PhD researcher
Arno Vanneste
PhD researcher
Jeroen Ooge
PhD researcher
Aditya Bhattacharya
PhD researcher
Ivania Donoso Guzmán
PhD researcher
4. Explainable Artificial Intelligence (XAI)
“Given an audience, an explainable artificial
intelligence is one that produces details or reasons to
make its functioning clear or easy to understand.”
[Arr20]
4
[Arr20] Arrieta, Alejandro Barredo, et al. "Explainable Artificial Intelligence (XAI): Concepts, taxonomies,
opportunities and challenges toward responsible AI." Information Fusion 58 (2020): 82-115.
5. Explaining model outcomes to increase user trust and acceptance
Enable users to interact with the explanation process to improve the model
Objectives
Models
12. Explanations
12
Millecamp, M., Htun, N. N., Conati, C., & Verbert, K. (2019, March). To explain or not to explain: the
effects of personal characteristics when explaining music recommendations. In Proceedings of the 2019
Conference on Intelligent User Interface (pp. 397-407). ACM.
13. Personal characteristics
Need for cognition
• Measurement of the tendency for an individual to engage in, and enjoy, effortful cognitive activities
• Measured by test of Cacioppo et al. [1984]
Visualisation literacy
• Measurement of the ability to interpret and make meaning from information presented in the form of
images and graphs
• Measured by test of Boy et al. [2014]
Locus of control (LOC)
• Measurement of the extent to which people believe they have power over events in their lives
• Measured by test of Rotter et al. [1966]
Visual working memory
• Measurement of the ability to recall visual patterns [Tintarev and Mastoff, 2016]
• Measured by Corsi block-tapping test
Musical experience
• Measurement of the ability to engage with music in a flexible, effective and nuanced way
[Müllensiefen et al., 2014]
• Measured using the Goldsmiths Musical Sophistication Index (Gold-MSI)
Tech savviness
• Measured by confidence in trying out new technology
13
14. User study
Within-subjects design: 105 participants recruited with Amazon Mechanical Turk
Baseline version (without explanations) compared with explanation interface
Pre-study questionnaire for all personal characteristics
Task: Based on a chosen scenario for creating a play-list, explore songs and rate all
songs in the final playlist
Post-study questionnaire:
Recommender effectiveness
Trust
Good understanding
Use intentions
Novelty
Satisfaction
Confidence
16. Design implications
Explanations should be personalised for different groups of end-
users.
Users should be able to choose whether or not they want to see
explanations.
Explanation components should be flexible enough to present
varying levels of details depending on a user’s preference.
16
17. User control
Users tend to be more satisfied when they have control over
how recommender systems produce suggestions for them
Control recommendations
Douban FM
Control user profile
Spotify
Control algorithm parameters
TasteWeights
18. Controllability Cognitive load
Additional controls may increase cognitive load
(Andjelkovic et al. 2016)
Ivana Andjelkovic, Denis Parra, andJohn O’Donovan. 2016. Moodplay: Interactive mood-based music
discovery and recommendation. In Proc. of UMAP’16. ACM, 275–279.
19. Different levels of user control
19
Level
Recommender
components
Controls
low Recommendations (REC) Rating, removing, and sorting
medium User profile (PRO)
Select which user profile data
will be considered by the
recommender
high
Algorithm parameters
(PAR)
Modify the weight of different
parameters
Jin, Y., Tintarev, N., & Verbert, K. (2018, September). Effects of personal characteristics on music recommender
systems with different levels of controllability. In Proceedings of the 12th ACM Conference on Recommender
Systems (pp. 13-21). ACM.
20. User profile (PRO) Algorithm parameters (PAR) Recommendations (REC)
8 control settings
No control
REC
PAR
PRO
REC*PRO
REC*PAR
PRO*PAR
REC*PRO*PAR
21. Evaluation method
Between-subjects – 240 participants recruited with AMT
Independent variable: settings of user control
2x2x2 factorial design
Dependent variables:
Acceptance (ratings)
Cognitive load (NASA-TLX), Musical Sophistication, Visual Memory
Framework Knijnenburg et al. [2012]
22. Results
Main effects: from REC to PRO to PAR → higher cognitive load
Two-way interaction: does not necessarily result in higher
cognitive load. Adding an additional control component to
PAR increases the acceptance. PRO*PAR has less cognitive
load than PRO and PAR
High musical sophistication leads to higher quality, and thereby
result in higher acceptance
22
Jin, Y., Tintarev, N., & Verbert, K. (2018, September). Effects of personal characteristics on music
recommender systems with different levels of controllability. In Proceedings of the 12th ACM Conference on
Recommender Systems (pp. 13-21). ACM.
27. Explaining exercise recommendations
How to automatically adapt
the exercise recommending
on Wiski to the level of
students?
How do (placebo)
explanations affect initial trust
in Wiski for recommending
exercises?
Goals and research questions
Automatic adaptation Explanations & trust Young target audience
Middle and high school students
Ooge, J., Kato, S., Verbert, K. (to appear) Explaining Recommendations in E-Learning: Effects on Adolescents'
Initial Trust. Proceedings of the 27th IUI conference on Intelligent User Interfaces
28. Methodology: Automatic adaptation
All Questions
Elo
Filter
Potential Questions
of Same Level
Rank with
Collaborative
Filtering
Sorted
Recommended
Questions
* topics are chosen by the user and are
thus not part of the recommendation
scheme
Combine Elo rating system with collaborative filtering:
• Elo rating system finds questions of similar difficulty level
• Collaborative filtering ranks found questions
29. Methodology: Explanations
Iterative design of explanation interfaces through a user-
centred design methodology
Full-fledged tutorial for full transparency Single-screen explanation Final explanation interface
33. Methodology: Trust
Multidimensional trust
• Trusting beliefs (Competence, benevolence,
integrity)
• Intention to return
• Perceived Transparency
One-dimensional trust
Direct trust measurement
Ask about trust factors
with 7-point Likert-type
questions
Indirect trust measurement
Log whether students accept
recommendations or not
34. Results: Real explanations…
… did increase multidimensional initial trust
… did not increase one-dimensional initial trust
… led to accepting more recommended exercises
compared to both placebo and no explanations
35. Results: Placebo explanations…
… did not increase initial trust compared to no explanations
… may undermine perceived integrity
… are a useful baseline:
• how critical are students towards explanations?
• how much transparency do students need?
36. Results: No explanations
Can be acceptable in low-stakes situations (e.g., drilling
exercises):
indications of difficulty level might suffice
Personal level indication:
Easy, Medium and Hard
tags
38. 38
uncertainty
Gutiérrez Hernández F., Seipp K., Ochoa X., Chiluiza K., De Laet T., Verbert K. (2018). LADA: A learning
analytics dashboard for academic advising. Computers in Human Behavior, pp 1-13. doi:
10.1016/j.chb.2018.12.004
LADA: a learning analytics dashboard for
study advisors
40. Results
LADA was perceived as a valuable tool for more accurate and
efficient decision making.
LADA enables expert advisers to evaluate significantly more
scenarios.
More transparency in the prediction model is required in order
to increase trust.
40
Gutiérrez Hernández F., Seipp K., Ochoa X., Chiluiza K., De Laet T., Verbert K. (2018). LADA: A learning
analytics dashboard for academic advising. Computers in Human Behavior, pp 1-13. doi:
10.1016/j.chb.2018.12.004
43. AHMoSe
Rojo, D., Htun, N. N., Parra, D., De Croon, R., & Verbert, K. (2021). AHMoSe: A knowledge-based visual
support system for selecting regression machine learning models. Computers and Electronics in Agriculture,
187, 106183.
45. Case Study – Grape Quality Prediction
45
Grape Quality Prediction Scenario [Tag14]
Data
Years 2010, 2011 (train) 2012 (test)
48 cells (Central Greece)
Knowledge-based rules
[Tag14] Tagarakis, A., et al. "A fuzzy inference system to model
grape quality in vineyards." Precision Agriculture 15.5 (2014):
555-578.
Source: [Tag14]
46. Simulation Study
AHMoSe vs full AutoML approach to support model selection.
46
RMSE (AutoML) RMSE (AHMoSe) Difference %
Scenario A
Complete
Knowledge
0.430 0.403 ▼ 6.3%
Scenario B
Incomplete
Knowledge
0.458 0.385 ▼ 16.0%
47. Qualitative Evaluation
10 open ended questions
5 viticulture experts and 4 ML experts.
Thematic Analysis: potential use cases, trust, usability, and
understandability.
48. Qualitative Evaluation - Trust
48
Showing the dis/agreement of model outputs with expert’s
knowledge can promote trust.
“The thing that makes us trust the models is the fact that most of
the time, there is a good agreement between the values
predicted by the model and the ones obtained for the knowledge
of the experts.”
– Viticulture Expert
51. Predicting duration to find a job
51
Key Issues: Missing data, prediction trust issues, job seeker
motivation, lack of control.
52. Methodology
A Customer Journey approach. (5 mediators).
Hands-on time with the original dashboard (22 mediators).
Observations of mediation sessions. (3 mediators, 6 job seekers).
Questionnaire regarding perception of the dashboard and prediction
model (15 Mediators).
52
Charleer S., Gutiérrez Hernández F., Verbert K. (2018). Supporting job mediator and job seeker through an
actionable dashboard. In: Proceedings of the 24th IUI conference on Intelligent User Interfaces Presented at
the ACM IUI 2019, Los Angeles, USA.
54. Evaluation
Qualitative evaluation with expert users:
(N = 12, 10f, age: M= 40.7, SD = 9.4)
Semi-structured interviews
1. Feedback on parameter visuals.
2. Interaction feedback with the working prototype dashboard.
54
55. Results
Our design attempts to clarify predictions by supporting
conversations between mediators and job seekers.
Need for customization and contextualization.
The human expert plays a crucial role when interpreting and
relaying in the predicted or recommended output.
Our explanatory tool helps mediators to control the message
they wish to convey depending on the situation context.
55
Charleer S., Gutiérrez Hernández F., Verbert K. (2019). Supporting job mediator and job seeker
through an actionable dashboard. In: Proceedings of the 24th IUI conference on Intelligent User
Interfaces Presented at the ACM IUI 2019, Los Angeles, USA. (Core: A)
56. Second dashboard: explaining
recommendations
56
Gutiérrez, F., Charleer, Sven, De Croon, Robin, Nyi Nyi Htun, Goetschalckx, Gerd, & Verbert, Katrien.
(2019) “Explaining and exploring job recommendations: a user-driven approach for interacting with
knowledge-based job recommender systems”. In Proceedings of the 13th ACM Conference on
Recommender Systems. ACM, 2019
60. Labor Market Explorer Design Goals
60
[DG1] Exploration/Control
Job seekers should be able to control
recommendations and filter out the information
flow coming from the recommender engine by
prioritizing specific items of interest.
[DG2] Explanations
Recommendations and matching scores should be
explained, and details should be provided on-
demand.
[DG3] Actionable Insights
The interface should provide actionable insights to
help job-seekers find new or more job
recommendations from different perspectives.
68. Results: user empowerment
The approach is perceived as effective to explore job
recommendations.
Most participants felt confident and will use the explorer again.
Explanations contribute to support user empowerment.
A diverse set of actionable insights were also mentioned by
participants.
68
69. Results: personal characteristics
The explorer was slightly better perceived by older participants
(45+).
Participants in the technical group engaged more with all the
different features of the dashboard.
Non-native speakers, sales and construction groups engaged
more with the map.
The table overview was perceived as very useful by all user
groups, but the interaction may need further simplification for
some users.
69
76. Design and Evaluation
76
Gutiérrez F., Cardoso B., Verbert K. (2017). PHARA: a personal health augmented reality assistant to support
decision-making at grocery stores. In: Proceedings of the International Workshop on Health Recommender
Systems co-located with ACM RecSys 2017 (Paper No. 4) (10-13).
77. Design
77
Gutiérrez Hernández F., Htun NN., Charleer S., De Croon R., Verbert K. (2018). Designing augmented reality
applications for personal health decision-making. In: Proceedings of the 2019 52nd Hawaii International
Conference on System Sciences Presented at the HICSS, Hawaii, 07 Jan 2019-11 Jan 2019.
79. Methodology
Within Subjects
n = 28 (1F, 27M) Ages from 22 to 38 (M = 25.81, SD = 4.57)
Post-Questionnaires
TAM (Technology Acceptance)
NASA-TLX (Task Load Index)
79
Gutiérrez Hernández F., Htun NN., Charleer S., De Croon R., Verbert K. (2018). Designing augmented reality
applications for personal health decision-making. In: Proceedings of the 2019 52nd Hawaii International
Conference on System Sciences Presented at the HICSS, Hawaii, 07 Jan 2019-11 Jan 2019.
80. Results
PHARA allows users to make informed decisions, and resulted
in selecting healthier food products.
Stack layout performs better with HMD devices with a limited
field of view, like the HoloLens, at the cost of some
affordances.
The grid and pie layouts performed better in handheld devices,
allowing to explore with more confidence, enjoyability and less
effort.
80
Gutiérrez Hernández F., Htun NN., Charleer S., De Croon R., Verbert K. (2018). Designing augmented
reality applications for personal health decision-making. In: Proceedings of the 2019 52nd Hawaii
International Conference on System Sciences Presented at the HICSS, Hawaii, 07 Jan 2019-11 Jan
2019.
85. Evaluation methodology
12 nurses used the app for three months
Data collection
Interaction logs
Resque questions
Semi-structured interviews
85
87. Results
Iterative design process identified several important features, such as the pending
list, overview and the feedback shortcut to encourage feedback.
Explanations seem to contribute well to better support the healthcare professionals.
Results indicate a better understanding of the call notifications by being able to see the
reasons of the calls.
More trust in the recommendations and increased perceptions of transparency and control
Interaction patterns indicate that users engaged well with the interface, although some
users did not use all features to interact with the system.
Need for further simplification and personalization.
87
93. Biofortification info
Plants to cultivate
PERNUG
Increased access to more nutritious plants
Improved iron and B12 intakes for vegan and vegetarian
subgroups
Consumer app with recipe recommendations Hydroponic system with
biofortified plants
https://www.eitfood.eu/projects/pernug
94.
95. Take-away messages
Involvement of end-users has been key to come up with
interfaces tailored to the needs of non-expert users
Actionable vs non-actionable parameters
Domain expertise of users and need for cognition important
personal characteristics
Need for personalisation and simpliciation
95
We focus specifically on visual analytics for non-expert users.
Non-expert users are defined as users that have little knowledge of data processing and analysis.
We research two algorithmic foundations: predictions models like regression and clustering and recommender systems that suggest items to users.
The key objective is to communicate the uncertainty of these models to support decision-making and increase trust.
We do it through the use of visualization techniques that explain the models.
Amazon.com gebruikt een collaborative filtering techniek: zoekt gelijkenissen tussen gebruikers en gaat dan op basis van wat gelijkaardige gebruikers kopen aanbevelingen doen.
The procedure contains the following steps:
\begin{enumerate}
\item \textit{Tutorial of study} - Participants were invited to read the description of the user study and to choose a scenario for generating a play-list. Then, they were asked to watch a task tutorial. Only the features of the particular setting were shown in this video. The ``Start'' button of the study was only activated after finishing the tutorial. Users logged in with their Spotify accounts to our experimental system, so that our recommenders could leverage the Spotify API and user listening history to generate ``real'' recommendations.
\item \textit{Pre-study questionnaire} - This questionnaire collects user demographics and measures user's personal characteristics such as musical sophistication and visual memory capacity. %and their trust in recommender systems.
The visual memory capacity is measured by ``Corsi block-tapping test''. In the test, a number of tiles are highlighted one at a time, and participants are asked to select the tiles in the correct order afterward. The number of highlighted tiles increases until the user makes too many errors. In Experiments 1 and 3, we used a test with a more sophisticated implementation of the Corsi test~\footnote{\url{https://www.humanbenchmark.com/tests/memory}, accessed June 2018}, which allows us to better distinguish participants by the level of visual memory capacity. In Experiment 2, to control the workload of participants in the within-subjects design, we chose a simple version of the Corsi test~\footnote{\url{http://www.psytoolkit.org/experiment-library/corsi.html}, accessed June 2018} for measuring visual short-term memory.
\end{itemize}
\item \textit{Manipulating Recommender and rating songs} - To ensure that participants spent enough time to explore recommendations, the questionnaire link was only activated after 10 minutes. After tweaking the recommender, participants were asked to rate the top-20 recommended songs that resulted from their interactions.
\item \textit{Post-study questionnaire} - Participants were asked to evaluate the perceived quality, perceived accuracy, perceived diversity, satisfaction, effectiveness, and choice difficulty of the recommender system. After answering all the questions, participants were given opportunities to provide free-text comments of their opinions and suggestions about our recommender.
\end{enumerate}
Figure 2 shows that participants with low NFC are reporting
a higher confident in their playlist with the explanations in-
terface than in the baseline. Participants with a high NFC
reported the opposite. Hence, the participants with low NFC
have more confidence in the explanation interface than in the
baseline, in contrast to user with low NFC. An explanation
might be that low NFC participants benefited from the expla-
nations because they did not spontaneously engage in much
extra reasoning to justify the recommendations they received,
and when they received the rational from the explanation this
increased their confidence in their songs selection.
Figure 2 also indicates that as the NFC increased, the con-
fidence of participants in the playlist created in the baseline
also increased. This result indicated that participants with a
high NFC were more willing to understand their own musical
preference in relation to the attributes of the recommended
songs. This may have resulted in a higher confidence in their
playlist.
We did not see the same increase in trust as NFC increases
in the explanation interface. As Figure 2 shows, the NFC
scores in the third quartile were almost the same for both
interfaces. At the highest NFC level, participants had a higher
confidence in the baseline than in the explanation interface.
The reduced confidence within the explanation interface could
be an indication that users with a high NFC have less need for
explanations.
We employed a between-subjects study to investigate the effects of interactions among different user control on acceptance, perceived diversity, and cognitive load. We consider each of three user control components as a variable. By following the 2x2x2 factorial design we created eight experimental settings (Table~\ref{tab:table1}), which allows us to analyze three main effects, three two-way interactions, and one three-way interaction. We also investigate which specific \textit{personal characteristics} (musical sophistication, visual memory capacity) influence acceptance and perceived diversity. Each experimental setting is evaluated by a group of participants (N=30). Of note, to minimize the effects of UI layout, all settings have the same UI and disable the unsupported UI controls, e.g., graying out sliders.
As shown in section ~\ref{evaluation questions}, we employed Knijnenburg et al.'s framework~\citep{knijnenburg2012explaining} to measure the six subjective factors, perceived quality, perceived diversity, perceived accuracy, effectiveness, satisfaction, and choice difficulty~\citep{knijnenburg2012explaining}. In addition, we measured cognitive load by using a classic cognitive load testing questionnaire, the NASA-TLX~\footnote{https://humansystems.arc.nasa.gov/groups/tlx}. It assesses the cognitive load on six aspects: mental demand, physical demand, temporal demand, performance, effort, and frustration.
The procedure follows the design outlined in the general methodology (c.f., Section \ref{sec:general-procedure}). The \textit{experimental task} is to compose a play-list for the chosen scenario by interacting with the recommender system. Participants were presented with play-list style recommendations (Figure~\ref{fig:vis1}c). Conditions were altered on a between-subjects basis. Each participant was presented with only one setting of user control. For each setting, initial recommendations are generated based on the selected top three artists, top two tracks, and top one genre. According to the controls provided in a particular setting, participants were able to manipulate the recommendation process.
Main effects: REC has lowest cgload and highest acceptance
Two-way: All the settings that combine two control components do not lead to significantly higher cognitive load than using only one control component. combing multiple control components potentially increases acceptance without increasing cognitive load significantly.
visual memory is not a significant factor that affects the cognitive load of controlling recommender systems. In other words, controlling the more advanced recommendation components in this study does not seem to demand a high visual memory. In addition, we did not find an effect of visual memory on acceptance (or perceived accuracy and quality).
One possible explanation is that users with higher musical so- phistication are able to leverage different control components to explore songs, and this influences their perception of recommenda- tion quality, thereby accepting more songs.
Our results show that the settings of user control significantly influence cognitive load and recommendation acceptance. We discuss the results by the main effects and interaction effects in a 2x2x2 factorial design.
Moreover, we discuss how visual memory and musical sophistication affect cognitive load, perceived diversity, and recommendation acceptance.
\subsubsection{Main effects}
We discuss the main effects of three control components. Increased control level; from control of recommendations (REC) to user profile (PRO) to algorithm parameters (PAR); leads to higher cognitive load (see Figure \ref{fig:margin}c). The increased cognitive load, in turn, leads to lower interaction times. Compared to the control of algorithm parameters (PAR) or user profile (PRO), the control of recommendations (REC) introduces the least cognitive load and supports users in finding songs they like.
We observe that most existing music recommender systems only allow users to manipulate the recommendation results, e.g., users provide feedback to a recommender through acceptance. However, the control of recommendations is a limited operation that does not allow users to understand or control the deep mechanism of recommendations.
\subsubsection{Two-way interaction effects}
Adding multiple controls allows us to improve on existing systems w.r.t. control, and do not necessarily result in higher cognitive load. Adding an additional control component to algorithm parameters increases the acceptance of recommended songs significantly.
Interestingly, all the settings that combine two control components do \textit{not} lead to significantly higher cognitive load than using only one control component. We even find that users' cognitive load is significantly \textit{lower} for (PRO*PAR) than (PRO, PAR), which shows a benefit of combining user profile and algorithm parameters in user control. Moreover, combing multiple control components potentially increases acceptance without increasing cognitive load significantly. Arguably, it is beneficial to combine multiple control components in terms of acceptance and cognitive load.
\subsubsection{Three-way interaction effects}
The interaction of PRO*PAR*REC tends to increase acceptance (see Figure \ref{fig:margin}a), and it does not lead to higher cognitive load (see Figure \ref{fig:margin}c). Moreover, it also tends to increase interaction times and accuracy. Therefore, we may consider having three control components in a system.
Consequently, we answer the research question. \textbf{RQ1}: \textit{The UI setting (user control, visualization, or both) has a significant effect on recommendation acceptance?} It seems that combining PAR with a second control component or combing three control components increases acceptance significantly. %KV: this paragraph refers to different RQs: either rephrase or omit? -SOLVED
\subsubsection{Effects of personal characteristics}
Having observed the trends across all users, we survey the difference in cognitive load and item acceptance due to personal characteristics. We study two kinds of characteristics: visual working memory and musical sophistication.
\paragraph{Visual working memory}
The SEM model suggests that visual memory is not a significant factor that affects the cognitive load of controlling recommender systems. The cognitive load for the type of controls used may not be strongly affected by individual differences in visual working memory. In other words, controlling the more advanced recommendation components in this study does not seem to demand a high visual memory.
In addition, we did not find an effect of visual memory on acceptance (or perceived accuracy and quality). Finally, the question items for diversity did not converge in our model, so we are not able to make a conclusion about the influence of visual working memory on diversity.
\paragraph{Musical sophistication}
Our results imply that high musical sophistication allows users to perceive higher recommendation quality, and may thereby be more likely to accept recommended items. However, higher musical sophistication also increases choice difficulty, which may negatively influence acceptance.
One possible explanation is that users with higher musical sophistication are able to leverage different control components to explore songs, and this influences their perception of recommendation quality, thereby accepting more songs. Finally, the question items for diversity did not converge in our model, so we are not able to make a conclusion about the influence of musical sophistication on diversity.
As I have already explained, this is the LADA Dashboard that predicts the chance of success and presents a set of components that are intended to help the student adviser to give feedback to the student.
We evaluated this dashboard with both laymen and experts. They used LADA based on real data of students to plan a semester for a student in two Conditions:
Using the dashboard.
Using the traditional system.
Results indicate that the prediction models enables users to explore more possible scenarios,
but more transparency is required.
The quality indicator is insuffient to increase user trust
The first application domain is agriculture
Precision agriculture is an interesting domain to research the representation of data and uncertainty associated with both data and prediction models for non-expert users, such as farmers.
This domain faces some typical challenges of Visual Analytics, missing data and uncertainty of predictions.
In this work, we conducted a systematic review of visualisation techniques and the representation of uncertainty.
The transcribed data were coded and analysed following the thematic analysis approach (Braun and Clarke, 2006), which resulted in four main themes: potential use cases, trust, usability, and understandability.
Marimekko charts difficult
Showing the dis/agreement of model outputs with expert’s knowledge can promote understandability and trust.
ability to see dis/agreements between models' predictions and an expert's knowledge can help them inspect further and thus promote trust.
u
Job recommender systems have become a well researched area. In this dissertation, we designed and evaluated two interactive dashboards that can help explain the reasoning behind job recommendations and predictions.
A first dashboard has been elaborated that explain predictions of the chance to find a job in a particular job area.
The second dashboard explains job recommendations by showing competences and competence gaps instead of the typical matching score used by recommender systems.
This is the first dashboard:
We designed this dashboard on top of a prediction model to explain the inner workings to job mediators.
We make a distinction between actionable and non-actionable parameters.
Age is an example of a non-actionable parameter.
We used a user-centered design methodology consisting of the steps listed on this slide.
This is the first dashboard:
We designed this dashboard on top of a prediction model to explain the inner workings to job mediators.
We make a distinction between actionable and non-actionable parameters.
Age is an example of a non-actionable parameter.
Job mediators highlighted the importance of customising the dashboard to be able to control the message.
Five mediators used negative parameters to support their message
Two mediators removed negative parameters to avoid demotivation.
“age can be demotivating”
“too much information might be difficult to process” “would like to see an overview of everything” “depends on the job seeker”
A second dashboard was designed for both job seekers and job mediators.
The dashboard explains recommendations by showing compentence and compentence gaps.
We used a user-centered design methodology consisting of:
focus groups,
co-design sessions,
usability evaluations
and a final evaluation with 66 job seekers.
The results are presented here: job title and distance to a job were ranked frequently on the first positiion by both job seekers and job mediators. Many other parameters were ranked frequently on the next positions, including type of contract, competences, studies and work experience.
We then elaborated designs that put focus on the important parameters: location and distance are for instace visualised in all tthese design, as well as competences that are asked in jobs. We captured additional input of job seekers and mediators in co-design sessions and evaluated first prototypes.
Results of these sessions were captured and articulated in the following design goals:
Exploration / control: the need to control recommendations and highlight items of interest was expressed
Job seekers also wanted to see explanations of recommendations that go beyong the typical matching score
There is also a need to support actionable insights: for instance by visualising which competences are currently in high demand in the job market
We used a user-centered design methodology consisting of:
focus groups,
co-design sessions,
usability evaluations
and a final evaluation with 66 job seekers.
We conducted a final evaluation with 66 job seekers, in different training programs at VDAB – including highly technical users, sales, construction and non-native speakers.
We asked the users to freely explore the dashboard and then used the ResQue questionnaire to capture user experience feedback together with two open questions.
We also logged all interactions.
These are the results of the ResQue questionnaire: overall the dashboard was well received – particularly also the explanations and ease of use.
Technical users were a bit more negative with respect to overall satisfaction (Q5).
Perceived accuracy (Q1) was a bit lower for non-native speakers: As language is not trivial for this group, also working with an interface in a different language as well as finding suitable jobs is evidently much more difficult.
Perceived usefulness (Q2) was a bit lower for the construction group. The wide range of very specific competencies in their domain was one of the reasons they highlighted for this lower score.
The explorer was slightly better perceived by older participants (45+) with respect to overall satisfaction (Q5) and use intention (Q4). Confidence (Q6) was slightly higher for younger participants.
To better understand the use of the tool, we logged the clicks of participants through the different visualization components. – including the vacancies map, the filters and the vacancies table
Participants with technical background engaged the most with the job vacancies table, where they performed most of the interactions in a significant different way compared to the other groups.
Participants in the construction group and non-native speakers engaged less with the job vacancies table. We can observe that these groups engaged more with the map component.
Responses to the questions indicate that job seekers value the use of interactive visualization techniques to find relevant job vacancies.. In general, the approach is perceived as effective to explore job recommendations. The interaction patterns also indicate that participants engaged well with the interface.
Also explanations seem to contribute well to better support of user empowerment. Many of the responses hint to better understanding of the job recommendations by being able to see which competencies are required.
A diverse set of actionable insights were also mentioned by participants. Participants indicate that the overview of competencies enables them to explore the job market and understand whether they have the needed competencies. They also gain insight into regions where most jobs are offered, supporting potential location-based job mobility. Other actionable insights include the observation that their user profile is not up to date. Such insights are key as well, as they trigger job seekers to update their profile and, in turn, receive better recommendations.
In general, we can observe only few differences with respect to the age and gender of different participants. The explorer was slightly better perceived by older participants who all indicated that the explorer is a good tool and that they would use the explorer again.
Health is another interesting domain: here we try to provide relevant information to end-users that is trustworthy and has a positive impact on decision-making.
We researched the representation of uncertainty of a prediction model that predicts the impact of a food product on weight as well as different layouts to present this data together with recommendations in an AR setting.
We found that the stack visualisation performs better with HMD devices with a limited field of view, like the HoloLens, at the cost of some usability affordances (RQ4).
For handheld devices, the grid and the pie tended to score higher in terms of confidence in decision making, compared to the list and stack layouts (RQ1, RQ4).
The prediction model shows the impact of the food product on the weight of the participant. Opacity is used to represent the uncertainty of this prediction. (POINT to third card)
The prediction model shows the impact of the food product on the weight of the participant. Opacity is used to represent the uncertainty of this prediction. (POINT to third card)
We compared four different layouts to represent this information:
a stack layout,
a list layout,
a grid layout and
a pie layout.
We compared their use in two implementations:
one using the Microsoft HoloLens,
and a second one using an Smartphone
We evaluated these layouts in a user study with 28 participants in a lab setting.
and measured both subjective and objective data collected from the use of our application,
We found that the stack visualisation performs better with HMD devices with a limited field of view, like the HoloLens, at the cost of some usability affordances (RQ4).
For handheld devices, the grid and the pie tended to score higher in terms of confidence in decision making, compared to the list and stack layouts (RQ1, RQ4).
Q11: not mych effort
Q2, Q5
During the interviews, some participants also appreciated the explanations of the system, indicating in particular the detailed view of the call, where they can see the most frequent reasons for a call: \say{\textit{\textbf{P8}: I like that you see the most frequent reasons. Why they call most often. That is the most important to me. That you can see it again afterward. Maybe, when a person calls a lot, that you can reflect. they go a lot to the toilet, maybe they have a urinary infection. When many nurses visit a resident, maybe you don’t see this if you don’t look at the overview.}}Overview of resuldents
Ik heb het verschil geplot van de scores wanneer ze textuele (blauw) of visuele (oranje) explanations gebruiken t.o.v. de hybride. Een score van TrustDiff 3 blauw wilt dus zeggen dat de gebruiker de hybride explanation 3 punten hoger geeft op trust t.o.v. de textuele explanation
Deze heb ik dan i.f.v de Need for Cognition en Ease of Satisfaction geplot van de gebruiker
Zo zien we bijvoorbeeld dat gebruikers die een hogere NFC hebben, de hybride explanations beter scoren i.t.v. Usefulness, transparency en Satisfaction vergeleken met de tekstuele explanations (edited)
Visueel t.o.v. hybride heeft juist een licht omgekeerd effect, hiervoor zou ik nog eens in de kwalitatieve logs moeten gaan kijken om dit te kunnen verklaren, lijkt mij een interessant gegeven