Google Sign Case Study Deck

Search with Sign Language
Google Sign
project 1: search redesigned
Melanie wells | wi 2023 | des 371b

Background

Those with hearing impairments, who use a sign language
as their primary language, experience language barriers
and challenges when using text-based search.
Problem

In other words, for signers, using text is like communicating
in an entirely different language, and creates frustrating
barriers to natural, intuitive articulation.
Problem

“Many deaf people don’t see
themselves as disabled,
they see themselves as a
cultural linguistic minority.”
“Sign languages are not simple
gestural codes based on spoken
language. They are four-
dimensional spatial languages that
do not have a written form, their
grammar and syntax are very
different from spoken languages.”
“The primary language of a
country, spoken language, is
for many deaf people their
second language — you can
compare deaf readers with
second language readers.”
Marie van Driessche, “Inclusive Design: Designing for Deaf People Helps Everyone”

Recent technological advancements in sign language video
translation have exposed an opportunity to make search more
accessible and intuitive by enabling gestural, video-based input.
Opportunity

How might we design an intuitive, fun, and effective search
experience based on gestures to...
Opportunity
...provide relief from the constraints of text-by-default search?

...unlock signers’ more intuitive modes of articulation?

...bring joy to the experience of searching with sign?

Organize the world’s information
and make it universally
accessible and useful.
Improve accessibility and intuitiveness
of the search experience for those who
are deaf or hearing impaired.
Mission Value Proposition

Live Query Translation

Sign Visualization

Multi-Modal Modification

Device Interoperability

Core Functionality
Gesture with Wearable

Widget with Live Camera

[Gesture with IoT] *TBD

[In-Video Plugin] *TBD

Initiatition

Prototypes

*Signs are for placement purposes only and
don’t reflect true translations

**Sign gifs by roxana Goncerzewicz; next step
for me is to design these myself, adhering to
google visual design principles
Tablet & Watch

iOS Widget
*Signs are for placement purposes only and
don’t reflect true translations

**Sign gifs by roxana Goncerzewicz; next step
for me is to design these myself, adhering to
google visual design principles

Primary inputs, outputs, and
visual feedback are sign first,
supported by other modes.
Gesture and interact across
integrated devices, making  
sign a touch, tap, or wave away.
Composing and modifying
queries is fun, responsive,
and engaging.
Sign First Seamless Magical

Process

Hypothesis How might we redesign search for sign language input, given recent technological advances in sign language video
retrieval and machine learning?
Google’s mission is to “organize the world’s information and make it universally accessible and useful” — however
accessibility features for the deaf community, including search by live/recorded sign language input, have been largely
unexplored and/or implemented.
Sign-based search would allow the deaf community to overcome the language barrier that exists with strictly text-based
input and return more accurate search results, improving Google’s core mission for accessibility in search.
Mission Statement
Value Proposition
UX Pillars
Attributes
Personalized

Content results are adapted to viewers
Meaningful

Build connections
Family-Friendly

Inspire the kid-at-heart
Inclusive

Friendly

Curious
Minimalist

Organized

Refined
Intuitive

Responsive

Adapible
behavior Appearance
Voice & tone
Framework

Considerations
Input Types
Live video-to-text translation, multi-modal annotation, phraseology and word organization visualization, multi-modal query
supplement/modification
How is search initiated? Search bar? Video call? Video content?

How can UI design help approach pain points around language barriers and language structure differences?

What are other relevant use cases that might inform the design specifically for the deaf community?

How might UI design approach the pain points found in language barriers?

How might additional modes of input supplement or modify search queries for more accurate results?
PRIMARY: Live or recorded video 
SECONDARY:Text and audio (and perhaps still image)
UI Concept
Questions
Pros & Cons
Challenges with working directly with
deaf community
Novel

Relevant

Based on Study User Needs
Pros Cons

Wireframes
Search
Translate Text Homework Shop
Google Lens
Retake
Thai
Green Lake.
restaurants
town
near
Search
Translate Text Homework Shop Search
Google Lens Google Lens
Concept #1
Live video-to-text translation
with phraseology
Search
Google Lens
Retake
Thai
Green Lake.
restaurants
town
near
Thai restaurants near Green Lake

Wireframes
00:03 00:03 00:03 00:03
Concept #2
Multi-modal annotation with
video scrubbing
Search
Search
Google Lens Google Lens Google Lens
Google Lens
Retake
Thai
Green Lake.
restaurants
town
near
Retake
Thai
Green Lake.
restaurants
town
near Thai
Green Lake.
cafés town
near
“cafés” Retake

Wireframes
Concept #3
Multi-modal query modification
post-return
Results Results
Best Match Best Match
Apps Apps
Other Section Other Section
Thai
Green Lake.
cafés town
near Thai
Green Lake.
cafés town
near
Search
Google Lens
one mile
Search
Google Lens
Retake
Replace with
?
near
one mile from

simple | playful | systematic
Universally Useful

informative | clean | sophisticated
Elegant Technicality

saturated | textural | Dark
Moody & Rich

Original Storyboard

LoFi Prototype

Demo Storyboard

Previous Concepts/Storyboards
Proposal 3: disney+ multi-modal search
10 Melanie wells | wi 2023 | des 371b
Input Types
Visual-forward, interactive UI that responds to the searcher
Characters, style, time period, collection, mood/feeling
UI Concept
Pros & Cons
Challenge on smartTV/console UI
Well-Known Existing Platform

Fun Content

Room to Improve!
Pros Cons
Considerations
Storyboard
Framework
Hypothesis How might we redesign search for multi-modal input to enhance the experience of searching for movies, shows, and
documentaries on the platform Disney+?
Disney+ is the home for streaming all Disney, Pixar, Marvel, Star Wars, and National Geographic content — its parent
company Disney, is a world-renown entertainment company driven to entertain, inform and inspire through storytelling.
Multi-modal search that taps into memory, nostalgia, connection to story, and characters will help viewers more deeply
connect to the content they watch, and build new memories with those they watch it with.
Mission Statement
Value Proposition
UX Pillars
Attributes
Personalized

Content results are adapted to viewers
Meaningful

Build connections
Family-Friendly

Inspire the kid-at-heart
Friendly

Inspiring

Inclusive
Immersive

Active

Rich

Fun

Intuitive

Inspirational
behavior Appearance
Voice & tone
Proposal 2: google scholar additive search
Input Types
Search “tray” that collects relevant inputs that combine to characterize the qualities of desired results. Results become inputs.
Documents, people, text, image, audio
UI Concept
Pros & Cons
Searchable content is text-based, which
makes moving away from text hard
Highly Relevant

Opportunity to Improve

Well-Known Existing Platform
Pros Cons
Considerations
Storyboard
Framework
Hypothesis How might we redesign search for multi-modal input to enhance the experience of finding relevant information
in scholarly literature on Google Scholar?
Google Scholar is the leading platform to help people find relevant scholarly work from a variety of disciplines and
sources. Intuitive multi-modal search will better meet searchers’ goals and build the capabilities of the platform.
Searching for relevant information in scholarly literature is both a divergent and convergent activity. An additive search
experience would allow searchers to collect a variety of inputs that combine to characterize desired results, leading to
more relevant, yet unique, informational results that enhance the depth of their research.
Concise, yet Detailed

Provide only what is helpful
Connective

Draw helpful, unexpected connections
Accessible

Works for searchers of all types
Mission Statement
Value Proposition
UX Pillars
Attributes
Informative

Scholarly

Helpful

Clean

Organized

Playful
Intuitive

Responsive

Flexible
behavior Appearance
Voice & tone
Considerations
Input Types
Sensory search based on literal or analogous characteristics of recipes/food/ingredients to return search results.
How might we initiate search from a photo or memory?

How can we make transitions between modes or input types feel natural?

How do we account for different perceptions of senses/thoughts/feelings?

How can we solve the issues of constrictive “natural” language inputs?

What feedback does the user see to let them know they can/can’t do something?

What feedback does the user see to let them know the system understands and is working, or not?

How to include concept of AI search for what and how? (“Tell me about Italy in the style of a rap song”)
Literal or analogous: visuals, sound, smell, taste, and feel (physical and emotional)
UI Concept
Questions
Pros & Cons
Proposal 1: America’s test kitchen experiential recipe search
Very abstract
Novel

Relevant

Human Connection
Pros Cons
Storyboard
Framework
Hypothesis How might we redesign search for multi-modal input in order to make recipe finding more engaging, inspiring, and experiential
on America’sTest Kitchen’s digital platform?
America’sTest Kitchen investigates the fundamental principles of of cooking and shares techniques, tools, and ingredients to
“help curious cooks become confident cooks.
” A more experiential search to searching for recipes will enhance their audience’s
connection with the foods they prepare.
Cooking and enjoying food are such sensory experiences, and are points of connection to our memories and to others people. A
more sensory experience of searching for recipes will enhance this connection, as well as build new ones.
Sensory

Put the human senses first
Inspiring

Encourage exploration
Meaningful

Deepen connections
Informative / Curious

Friendly

Encouraging

Authentic

Warm

Bold

Customizable

Responsive

Exploratory
Mission Statement
Value Proposition
UX Pillars
Attributes
behavior Appearance
Voice & tone

Thank you.

Google Sign Case Study Deck

Recommended

Recommended

More Related Content

Similar to Google Sign Case Study Deck

Similar to Google Sign Case Study Deck (20)

Recently uploaded

Recently uploaded (20)

Google Sign Case Study Deck