1. It looks like you’re trying to
make a user interface for AI.
Do you need assistance?
The UI of AI
Prof. Lydia Chilton
chilton@cs.columbia.edu
Computer Science Department

2. People interact with systems to accomplish a goal
Set the room temperatureSend an email Find information

3. Good technology is often invisible.
Technology is best when it helps users
achieve their goal without drawing attention
to the technology itself.
Technology that requires users to remember
low-level details and execute many functions
are difficult to use.

4. Bad AI Interfaces
Aggressive interfaces that the draw
attention to themselves. This causes
users to lose focus on their goal.
Making decisions for you with a poor
understanding of your goals and context.

5. Good AI interfaces are aligned with your goals
Spam Detection in your inbox
Spam detection in your inbox
(that you never notice)
So you focus on your emails tasks.
Text prediction in email,
Triggers automatically.
User accepts or rejects prediction
The default is to ignore it.
Object recognition on the phone
(for blind/low vision people)
Easy to pull out the phone and
snap a photo
Invisible Ask for confirmation Used out in the world,
in context

6. AI is about pattern recognition
Label data Predict data
Transform data

7. Now is the time to shape what goals
AI helps us accomplish
• How can AI help us? In the following examples of User Interfaces for
AI, think about how similar problems or solutions could be applied in
your domain.
• The more ideas the better. For every 20 ideas, one is probably
possible to do. Don’t get discouraged, just have more ideas. You never
know what is possible for institutions to implement.
• Share you ideas widely. Consider posting them on twitter, or a blog.
Contact local universities with students looking for programming
projects.

8. Interfaces that use labeling AI
Image labeling:
embedded in social networks,
Aligned with your goal,
Asks for confirmation
Text labeling:
Embedded in your inbox
Organizes email into:
spam, importance, reply?
Easy to dismiss or undo
Help you stay organized

9. Interfaces that use prediction AI
Predicting Images:
Take a photo and see yourself
projected 20 years older
(sell skin crème and retirement
planning)
Predicting Text:
Embedded in writing tasks.
UIs present conservative suggestions.
The default is to not accept them
Predicting Numbers:
Embedded in recommender systems,
And decision support software
(Ticket prices, criminal sentencing, offering loans)
Take a photo
You in 20
years
Help you made decisions

10. Interfaces that use data transformation AI
Images to Text Language -> language Voice to Text Numbers to Text
Mobile UI –
to translate where you are. Easy data entry. Turn data into
readable stories
Text is easier to process
than images.
Put data in the right format so you can use it.

11. AI applications on mobile, wearable, home and
vehicle platforms extend the contexts of use.
Home devices
Mobile
Driving assistance –
parking, directions, health/stress monitoring
Help you use achieve goals within a context

12. AI is imperfect. It needs people to guide it’s
application to real problems.
Carrie J. Cai, et al. 2019. Human Centered Tools for Coping with Imperfect Algorithms During Medical Decision
Making. CHI 2019.

13. Summary

14. People interact with systems to accomplish a goal
Set the room temperatureSend an email Find information

15. Good technology is often invisible.
Technology is best when it helps users
achieve their goal without drawing attention
to the technology itself.
Technology that requires users to remember
low-level details and execute many functions
are difficult to use.

16. Bad AI Interfaces draw attention to themselves
and away from the goal
Aggressive interfaces that the draw
attention to themselves. This causes
users to lose focus on their goal.
Making decisions for you with a poor
understanding of your goals and context.

17. Good AI interfaces are aligned with your goals
Spam Detection in your inbox
Spam detection in your inbox
(that you never notice)
So you focus on your emails tasks.
Text prediction in email,
Triggers automatically.
User accepts or rejects prediction
The default is to ignore it.
Object recognition on the phone
(for blind/low vision people)
Easy to pull out the phone and
snap a photo
Invisible Ask for confirmation Used out in the world,
in context

18. Now is the time to shape what goals
AI helps us accomplish
• How can AI help us? In the following examples of User Interfaces for
AI, think about how similar problems or solutions could be applied in
your domain.
• The more ideas the better. For every 20 ideas, one is probably
possible to do. Don’t get discouraged, just have more ideas. You never
know what is possible for institutions to implement.
• Share you ideas widely. Consider posting them on twitter, or a blog.
Contact local universities with students looking for programming
projects.

19. What goals can AI help people accomplish?
• What goals could be aided by pattern recognition:
• Cognition: organization, decision making, data formatting, searching data
• Task: labeling, predicting, or transforming data.
• User: Patients, technicians, doctors, nurses, administrators
• Location: Home, office, hospital, mobile
• Device: desktop, mobile, wearable, car, home
• What feedback do you need to guide it?