SlideShare a Scribd company logo
1 of 18
Semiotics of Interaction:
Towards a UI Alphabet
Jan Brejcha, Charles University, Prague
Aaron Marcus, Aaron Marcus and Associates, Inc., Berkeley
HCII 2013, Las Vegas 26 July 2013
Semiotic Foundation
• Semiotics: theory of signs
• Sign: “something that stands for someone or some-thing in some respect or
capacity” (C.S. Peirce)
• Four semiotic dimensions:
• Pragmatic: psychological, biological, and sociological impacts of using
signs
• Semantic: meaning, reference, or action of signs
• Syntactic: relations among signs (syntax)
• Lexical: physical constraints of generating signs
• Pragmatics regarding UI analysis:
user-UI and user-UI-designer relationships
UI Language Components
• UI grammar is composed of basic elements: interaction sentence,
interaction games, rhetorical tropes, interaction phases, and patterns. The
grammar elements concern both the noun and verb phrase of a sentence.
• Discrete elements are the smallest elements to have a meaning.
• Interaction sentence is a meaningful unit describing a task in a user's
interaction. A set of interaction sentences with the same goal form an
interaction game.
• Narrative in UI is made both by the designer’s meta-communication and the
temporal aspect of perceiving UI elements.
• Rhetorical tropes are devices of persuasion and emphasis, often
presented as metaphors.
• Patterns are typical configurations of UI language components in different
settings
UI Language Components: Summary
• Grammar sets rules regarding interaction chains:
• What can be chained in a cause-effect interaction unit
• How it can be chained
• For what purpose it can be chained.
• Studying interaction grammar helps to improve UI and interaction
consistency
• By promoting consistency, interaction grammar helps to build
expectations
• Constraints reinforce interaction grammar rules
• Physical (affordances), logical (reasoning), and cultural (conventions)
UI Analysis and Annotation Carried Out in
Study
• To analyze visible and interaction relations in set of UIs, one must extract
intrinsic grammar forming the UI “language”
• Semiotic analysis uses structured interaction transcript based on interaction
sentences
• Resulting transcript serves as input for follow-on expert evaluation
Evaluation Methods
• Expert evaluation methods available:
• Cognitive walkthrough
• Heuristic evaluation (HE, our choice)
• Expert inspection
• Semiotic analysis (SA, our choice)
• Evaluation process of initial study:
• Study used two complex graphic design applications:
• Adobe Photoshop
• GIMP
• Evaluate two UIs using both HE and SA methods
• Compare data resulting from each
Heuristic Evaluation (HE)
• Semiotic analysis focuses on signs in the UI, extracts underlying codes that
make/break UI’s meaning.
• Primary objective:
• Establish underlying conventions
• Identify significant differences and oppositions
• Model system categories, relations, connotations, distinctions and rules of
combination employed. (Chandler)
• Benefits:
• Provides more insight, captures more compatible data, at lower cost
• More amendable to cross-cultural analysis
• More accessible/understandable to stakeholders
• Much previous data and analysis examples exist
• Low overlap with heuristic evaluation
Heuristic Evaluation: Criteria
• Aesthetic integrity and minimalist design
• Consistency and standards
• Direct manipulation/See and point
• Error prevention
• Feedback/Visible system status
• Fitt’s Law
• Flexibility and efficiency of use
• Help and documentation
• Help users recognize, diagnose, and recover from errors
• Information legibility/Density
• Match between system and real world
• Modelessness
• Perceived stability
• Recognition rather than recall
• User control and freedom
• Visible interfaces/WYSIWYG
Semiotic Analysis (SA)
• Semiotic analysis focuses on signs in the UI, extracts underlying codes that
make/break UI’s meaning.
• Primary objective:
• Establish underlying conventions
• Identify significant differences and oppositions
• Model system categories, relations, connotations, distinctions and rules of
combination employed. (Chandler)
• Benefits:
• Provides more insight, captures more compatible data, at lower cost
• More amendable to cross-cultural analysis
• More accessible/understandable to stakeholders
• Much previous data and analysis examples exist
• Low overlap with heuristic evaluation
Semiotic Analysis: Heuristics
• Actors, audience and paradigm. Defining who the UI users are, who the
intended audience of the UI is and what the leading interaction paradigm is.
• Symbols. Different kinds of symbols connote different semantic spaces, cultural
backgrounds and address different audiences.
• Syntax. Signs should be used in any given context only once and should not be in
conflict with its context.
• Rhetorical tropes. The most common rhetorical tropes in the UI are devices of
substitution: metaphor, metonymy, prosopopoeia, and synecdoche.
• Interaction phases. The interaction should form meaningful temporal units.
• Patterns. UI language components form different kinds of patterns.
UI Corpus: Selection for Initial Study
• To get a workable UI set, study must select sample body of material: UI
corpus
• Study compared UI of Adobe Photoshop CS2 and GIMP 2.6.7, both running
on Mac OS X 10.5.8.
• Corpus must be as homogeneous as possible (in substance and time)
• Study chose features that were commercially advertised, as well as basic
core functions which all users are likely to perform
UI Corpus: Functions Analyzed
• Study found five shared functions in both UIs:
• Reduce red-eye: One-click red-eye tool vs. Red-eye removal
• Barrel distortion: Optical lens correction vs. Perspective clone tool
• Eliminate an object: Healing brush vs. Clone tool
• Clone objects in perspective: Vanishing pt. vs. Perspective clone tool
• Customize UI: Keyboard shortcuts and menu customization vs. Configure
keyboard shortcuts
• Study process analyzed these functions:
• Transcribed step-by-step process involved (interaction sentences)
• Carried out HE and SA for each UI
UI Corpus: Interaction Sentence Example
for Adobe Photoshop, 1/2
• (0) Open picture to adjust
• (1) Find proper function in menu or tool palette
• (a) Sub-task: look in toolbox for button resembling intended action
• Nothing like that was found
• (b) Alternate sub-task: look through menu items (especially in what seemed
most related (Image -> Adjustments, and Filter) for relative command
• Found under Filter -> Vanishing point…
• Window called “Vanishing Point” appears
• Window shows live preview, “Create Plane Tool“, “Clone Tool", etc.
• (2) Click four corners according to information provided:
• “Click the four corners of a perspective plane or object in the image to
create an editing plane. Tear off perpendicular planes from the stretch
nodes of existing…)”
UI Corpus: Interaction Sentence Example
for Adobe Photoshop, 2/2
• (3) Select Clone tool
• (4) Option-click in plane to set
source
• Opt+click in plane to set source
point for clone
• Once source point is set,
click+drag to paint or clone
• Shift+click to extend stroke to last
click
• (5) Click-drag (to paint) several
times to clone in perspective
• See proposed results in preview.
• (6) Click “OK“ to apply changes
• (7) Save changes to file
UI Corpus: Results of HE vs SA
• HE results:
• More concise; of 16 heuristics, only small number applied each time
• Could be used not only on interaction-sentence level, but also as general
analysis of entire UI
• SA results:
• SA tended to be more verbose
• All elements could be applied every time
• Could seem repetitive in some instances
• Provided solid context of analysis
• Enables comparison of UIs from different time-periods to show changes in
paradigm, syntax, and related SA criteria
Summary
• We applied the semiotic and linguistic theories to an UI corpus of graphic manipulation
applications. At the same time we analyzed the corpus using heuristic evaluation.
• We presented the transcript of interaction sentences forming language games, that
served together with the actual UI as a basis for Heuristic evaluation (HE) and Semiotic
analysis (SA). Moreover, the transcript served as input for defining the different elements
involved in the interaction language.
• The SA provided the expected kind of data, that gathered a wider context than those from
HE. That said, SA can be used to complement the widely used expert evaluation
methods, but could possibly be defined to have a higher overlap with HE. In the latter
case, SA would need to be evaluated hand in hand with the interaction sentences.
• In summary, our study demonstrated the depth of investigation and breadth of insight that
SA can achieve in HCI and how this could enhance the current practice. Both methods
could be merged to provide a best-of-both solution.
Future Work Plan
• Develop a pattern visualization for UI language elements. Such visualization
would help quickly compare interaction structures with different kinds of UI in
one culture, or between two or more cultures.
• Compare the SA with other semiotic methods.
• Focus SA to UI pragmatics (persuasion, ideologies).
Semiotics of Interaction:
Towards a UI Alphabet
Jan Brejcha, Doctoral Candidate
Charles University in Prague
U Kříže 8, 15800 Praha 5 – Jinonice, Czech Republic
Tel: +420 602 277 588
Email: Jan@Brejcha.name
Web: Jan.Brejcha.name
Aaron Marcus, President
Aaron Marcus and Associates, Inc. (AM+A)
1196 Euclid Avenue, Suite 1F. Berkeley, California 94708-1640, USA
Tel: +1-510-601-0994, Fax: +1-510-527-1994
Email: Aaron.Marcus@AMandA.com
Web: www.AMandA.com

More Related Content

Similar to Semiotics of Interaction: Towards a UI Alphabet

User Centered Design Patterns and Related Issues – A Review
User Centered Design Patterns and Related Issues – A ReviewUser Centered Design Patterns and Related Issues – A Review
User Centered Design Patterns and Related Issues – A ReviewWaqas Tariq
 
Dr. Elizabeth Churchill, Google. Creating consumer grade developer experiences
Dr. Elizabeth Churchill, Google. Creating consumer grade developer experiencesDr. Elizabeth Churchill, Google. Creating consumer grade developer experiences
Dr. Elizabeth Churchill, Google. Creating consumer grade developer experiencesIT Arena
 
Presentation at Seminar I: Rhetorical Evaluation of User Interfaces
Presentation at Seminar I: Rhetorical Evaluation of User Interfaces Presentation at Seminar I: Rhetorical Evaluation of User Interfaces
Presentation at Seminar I: Rhetorical Evaluation of User Interfaces Omar Sosa-Tzec
 
Co-Extracting Opinions from Online Reviews
Co-Extracting Opinions from Online ReviewsCo-Extracting Opinions from Online Reviews
Co-Extracting Opinions from Online ReviewsEditor IJCATR
 
Open Source Design Pattern Library, Spreading Communities Thick: Open Source ...
Open Source Design Pattern Library, Spreading Communities Thick: Open Source ...Open Source Design Pattern Library, Spreading Communities Thick: Open Source ...
Open Source Design Pattern Library, Spreading Communities Thick: Open Source ...Allison Bloodworth
 
Topic 3 Human Computer Interaction
Topic 3 Human Computer InteractionTopic 3 Human Computer Interaction
Topic 3 Human Computer Interactionnur ezzaty
 
How to Embed UX Thinking in Your Next API
How to Embed UX Thinking in Your Next APIHow to Embed UX Thinking in Your Next API
How to Embed UX Thinking in Your Next APIPronovix
 
Tools of the UX Trade
Tools of the UX TradeTools of the UX Trade
Tools of the UX Tradedpanarelli
 
Advanced Methods for User Evaluation in Enterprise AR
Advanced Methods for User Evaluation in Enterprise ARAdvanced Methods for User Evaluation in Enterprise AR
Advanced Methods for User Evaluation in Enterprise ARMark Billinghurst
 
Facilitating Dialogue - Using Semantic Web Technology for eParticipation
Facilitating Dialogue - Using Semantic Web Technology for eParticipationFacilitating Dialogue - Using Semantic Web Technology for eParticipation
Facilitating Dialogue - Using Semantic Web Technology for eParticipationIMC Technologies
 
Auto Mapping Texts for Human-Machine Analysis and Sensemaking
Auto Mapping Texts for Human-Machine Analysis and SensemakingAuto Mapping Texts for Human-Machine Analysis and Sensemaking
Auto Mapping Texts for Human-Machine Analysis and SensemakingShalin Hai-Jew
 
Designing Gestural Interaction: Challenges and Pitfalls
Designing Gestural Interaction: Challenges and PitfallsDesigning Gestural Interaction: Challenges and Pitfalls
Designing Gestural Interaction: Challenges and PitfallsJean Vanderdonckt
 
Rhetorical Evaluation of User Interfaces
Rhetorical Evaluation of User InterfacesRhetorical Evaluation of User Interfaces
Rhetorical Evaluation of User InterfacesOmar Sosa-Tzec
 
User Experience Testing
User Experience TestingUser Experience Testing
User Experience TestingKarl Smith
 

Similar to Semiotics of Interaction: Towards a UI Alphabet (20)

User Centered Design Patterns and Related Issues – A Review
User Centered Design Patterns and Related Issues – A ReviewUser Centered Design Patterns and Related Issues – A Review
User Centered Design Patterns and Related Issues – A Review
 
Dr. Elizabeth Churchill, Google. Creating consumer grade developer experiences
Dr. Elizabeth Churchill, Google. Creating consumer grade developer experiencesDr. Elizabeth Churchill, Google. Creating consumer grade developer experiences
Dr. Elizabeth Churchill, Google. Creating consumer grade developer experiences
 
Presentation at Seminar I: Rhetorical Evaluation of User Interfaces
Presentation at Seminar I: Rhetorical Evaluation of User Interfaces Presentation at Seminar I: Rhetorical Evaluation of User Interfaces
Presentation at Seminar I: Rhetorical Evaluation of User Interfaces
 
Benoit Visual Only Retrieval
Benoit Visual Only RetrievalBenoit Visual Only Retrieval
Benoit Visual Only Retrieval
 
A brief overview of ux research
A brief overview of ux researchA brief overview of ux research
A brief overview of ux research
 
Co-Extracting Opinions from Online Reviews
Co-Extracting Opinions from Online ReviewsCo-Extracting Opinions from Online Reviews
Co-Extracting Opinions from Online Reviews
 
Open Source Design Pattern Library, Spreading Communities Thick: Open Source ...
Open Source Design Pattern Library, Spreading Communities Thick: Open Source ...Open Source Design Pattern Library, Spreading Communities Thick: Open Source ...
Open Source Design Pattern Library, Spreading Communities Thick: Open Source ...
 
Topic 3 Human Computer Interaction
Topic 3 Human Computer InteractionTopic 3 Human Computer Interaction
Topic 3 Human Computer Interaction
 
How to Embed UX Thinking in Your Next API
How to Embed UX Thinking in Your Next APIHow to Embed UX Thinking in Your Next API
How to Embed UX Thinking in Your Next API
 
Tools of the UX Trade
Tools of the UX TradeTools of the UX Trade
Tools of the UX Trade
 
ICT L4.pptx
ICT L4.pptxICT L4.pptx
ICT L4.pptx
 
Advanced Methods for User Evaluation in Enterprise AR
Advanced Methods for User Evaluation in Enterprise ARAdvanced Methods for User Evaluation in Enterprise AR
Advanced Methods for User Evaluation in Enterprise AR
 
Facilitating Dialogue - Using Semantic Web Technology for eParticipation
Facilitating Dialogue - Using Semantic Web Technology for eParticipationFacilitating Dialogue - Using Semantic Web Technology for eParticipation
Facilitating Dialogue - Using Semantic Web Technology for eParticipation
 
Auto Mapping Texts for Human-Machine Analysis and Sensemaking
Auto Mapping Texts for Human-Machine Analysis and SensemakingAuto Mapping Texts for Human-Machine Analysis and Sensemaking
Auto Mapping Texts for Human-Machine Analysis and Sensemaking
 
Icpc 2011 storey
Icpc 2011 storeyIcpc 2011 storey
Icpc 2011 storey
 
Designing Gestural Interaction: Challenges and Pitfalls
Designing Gestural Interaction: Challenges and PitfallsDesigning Gestural Interaction: Challenges and Pitfalls
Designing Gestural Interaction: Challenges and Pitfalls
 
Rhetorical Evaluation of User Interfaces
Rhetorical Evaluation of User InterfacesRhetorical Evaluation of User Interfaces
Rhetorical Evaluation of User Interfaces
 
Abstractive Review Summarization
Abstractive Review SummarizationAbstractive Review Summarization
Abstractive Review Summarization
 
User Experience Testing
User Experience TestingUser Experience Testing
User Experience Testing
 
Interaction-design-basic.pptx
Interaction-design-basic.pptxInteraction-design-basic.pptx
Interaction-design-basic.pptx
 

More from Jan Brejcha

Ideologies in HCI: A Semiotic Perspective
Ideologies in HCI: A Semiotic PerspectiveIdeologies in HCI: A Semiotic Perspective
Ideologies in HCI: A Semiotic PerspectiveJan Brejcha
 
Mezikulturní UX/UI očima sémiotiky: Jak designovat pro mimozemšťany?
Mezikulturní UX/UI očima sémiotiky: Jak designovat pro mimozemšťany?Mezikulturní UX/UI očima sémiotiky: Jak designovat pro mimozemšťany?
Mezikulturní UX/UI očima sémiotiky: Jak designovat pro mimozemšťany?Jan Brejcha
 
Hackers Congress v Paralelní Polis 17: „Liberate“, aneb za finanční svobodou ...
Hackers Congress v Paralelní Polis 17: „Liberate“, aneb za finanční svobodou ...Hackers Congress v Paralelní Polis 17: „Liberate“, aneb za finanční svobodou ...
Hackers Congress v Paralelní Polis 17: „Liberate“, aneb za finanční svobodou ...Jan Brejcha
 
Is there a token for that? Tokens demystified.
Is there a token for that? Tokens demystified.Is there a token for that? Tokens demystified.
Is there a token for that? Tokens demystified.Jan Brejcha
 
Chinese UI design guidelines 2.0
Chinese UI design guidelines 2.0Chinese UI design guidelines 2.0
Chinese UI design guidelines 2.0Jan Brejcha
 
Financial literacy in China as an innovation opportunity
Financial literacy in China as an innovation opportunityFinancial literacy in China as an innovation opportunity
Financial literacy in China as an innovation opportunityJan Brejcha
 
Innovation potential of the blockchain, and of decentralized applications
Innovation potential of the blockchain, and of decentralized applicationsInnovation potential of the blockchain, and of decentralized applications
Innovation potential of the blockchain, and of decentralized applicationsJan Brejcha
 
Decentralized Autonomous Organizations: Concept & Practical Examples
Decentralized Autonomous Organizations: Concept & Practical ExamplesDecentralized Autonomous Organizations: Concept & Practical Examples
Decentralized Autonomous Organizations: Concept & Practical ExamplesJan Brejcha
 
Jazyk uzivatelskeho rozhrani
Jazyk uzivatelskeho rozhraniJazyk uzivatelskeho rozhrani
Jazyk uzivatelskeho rozhraniJan Brejcha
 
Semiotika: Nastroj pro lepsi uzivatelsky prozitek?
Semiotika: Nastroj pro lepsi uzivatelsky prozitek?Semiotika: Nastroj pro lepsi uzivatelsky prozitek?
Semiotika: Nastroj pro lepsi uzivatelsky prozitek?Jan Brejcha
 
Zaklady interakcniho designu 2007 - Semiotika
Zaklady interakcniho designu 2007 - SemiotikaZaklady interakcniho designu 2007 - Semiotika
Zaklady interakcniho designu 2007 - SemiotikaJan Brejcha
 
Zaklady interakcniho designu 2007 - Zkusenost
Zaklady interakcniho designu 2007 - ZkusenostZaklady interakcniho designu 2007 - Zkusenost
Zaklady interakcniho designu 2007 - ZkusenostJan Brejcha
 

More from Jan Brejcha (12)

Ideologies in HCI: A Semiotic Perspective
Ideologies in HCI: A Semiotic PerspectiveIdeologies in HCI: A Semiotic Perspective
Ideologies in HCI: A Semiotic Perspective
 
Mezikulturní UX/UI očima sémiotiky: Jak designovat pro mimozemšťany?
Mezikulturní UX/UI očima sémiotiky: Jak designovat pro mimozemšťany?Mezikulturní UX/UI očima sémiotiky: Jak designovat pro mimozemšťany?
Mezikulturní UX/UI očima sémiotiky: Jak designovat pro mimozemšťany?
 
Hackers Congress v Paralelní Polis 17: „Liberate“, aneb za finanční svobodou ...
Hackers Congress v Paralelní Polis 17: „Liberate“, aneb za finanční svobodou ...Hackers Congress v Paralelní Polis 17: „Liberate“, aneb za finanční svobodou ...
Hackers Congress v Paralelní Polis 17: „Liberate“, aneb za finanční svobodou ...
 
Is there a token for that? Tokens demystified.
Is there a token for that? Tokens demystified.Is there a token for that? Tokens demystified.
Is there a token for that? Tokens demystified.
 
Chinese UI design guidelines 2.0
Chinese UI design guidelines 2.0Chinese UI design guidelines 2.0
Chinese UI design guidelines 2.0
 
Financial literacy in China as an innovation opportunity
Financial literacy in China as an innovation opportunityFinancial literacy in China as an innovation opportunity
Financial literacy in China as an innovation opportunity
 
Innovation potential of the blockchain, and of decentralized applications
Innovation potential of the blockchain, and of decentralized applicationsInnovation potential of the blockchain, and of decentralized applications
Innovation potential of the blockchain, and of decentralized applications
 
Decentralized Autonomous Organizations: Concept & Practical Examples
Decentralized Autonomous Organizations: Concept & Practical ExamplesDecentralized Autonomous Organizations: Concept & Practical Examples
Decentralized Autonomous Organizations: Concept & Practical Examples
 
Jazyk uzivatelskeho rozhrani
Jazyk uzivatelskeho rozhraniJazyk uzivatelskeho rozhrani
Jazyk uzivatelskeho rozhrani
 
Semiotika: Nastroj pro lepsi uzivatelsky prozitek?
Semiotika: Nastroj pro lepsi uzivatelsky prozitek?Semiotika: Nastroj pro lepsi uzivatelsky prozitek?
Semiotika: Nastroj pro lepsi uzivatelsky prozitek?
 
Zaklady interakcniho designu 2007 - Semiotika
Zaklady interakcniho designu 2007 - SemiotikaZaklady interakcniho designu 2007 - Semiotika
Zaklady interakcniho designu 2007 - Semiotika
 
Zaklady interakcniho designu 2007 - Zkusenost
Zaklady interakcniho designu 2007 - ZkusenostZaklady interakcniho designu 2007 - Zkusenost
Zaklady interakcniho designu 2007 - Zkusenost
 

Recently uploaded

SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 

Recently uploaded (20)

SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 

Semiotics of Interaction: Towards a UI Alphabet

  • 1. Semiotics of Interaction: Towards a UI Alphabet Jan Brejcha, Charles University, Prague Aaron Marcus, Aaron Marcus and Associates, Inc., Berkeley HCII 2013, Las Vegas 26 July 2013
  • 2. Semiotic Foundation • Semiotics: theory of signs • Sign: “something that stands for someone or some-thing in some respect or capacity” (C.S. Peirce) • Four semiotic dimensions: • Pragmatic: psychological, biological, and sociological impacts of using signs • Semantic: meaning, reference, or action of signs • Syntactic: relations among signs (syntax) • Lexical: physical constraints of generating signs • Pragmatics regarding UI analysis: user-UI and user-UI-designer relationships
  • 3. UI Language Components • UI grammar is composed of basic elements: interaction sentence, interaction games, rhetorical tropes, interaction phases, and patterns. The grammar elements concern both the noun and verb phrase of a sentence. • Discrete elements are the smallest elements to have a meaning. • Interaction sentence is a meaningful unit describing a task in a user's interaction. A set of interaction sentences with the same goal form an interaction game. • Narrative in UI is made both by the designer’s meta-communication and the temporal aspect of perceiving UI elements. • Rhetorical tropes are devices of persuasion and emphasis, often presented as metaphors. • Patterns are typical configurations of UI language components in different settings
  • 4. UI Language Components: Summary • Grammar sets rules regarding interaction chains: • What can be chained in a cause-effect interaction unit • How it can be chained • For what purpose it can be chained. • Studying interaction grammar helps to improve UI and interaction consistency • By promoting consistency, interaction grammar helps to build expectations • Constraints reinforce interaction grammar rules • Physical (affordances), logical (reasoning), and cultural (conventions)
  • 5. UI Analysis and Annotation Carried Out in Study • To analyze visible and interaction relations in set of UIs, one must extract intrinsic grammar forming the UI “language” • Semiotic analysis uses structured interaction transcript based on interaction sentences • Resulting transcript serves as input for follow-on expert evaluation
  • 6. Evaluation Methods • Expert evaluation methods available: • Cognitive walkthrough • Heuristic evaluation (HE, our choice) • Expert inspection • Semiotic analysis (SA, our choice) • Evaluation process of initial study: • Study used two complex graphic design applications: • Adobe Photoshop • GIMP • Evaluate two UIs using both HE and SA methods • Compare data resulting from each
  • 7. Heuristic Evaluation (HE) • Semiotic analysis focuses on signs in the UI, extracts underlying codes that make/break UI’s meaning. • Primary objective: • Establish underlying conventions • Identify significant differences and oppositions • Model system categories, relations, connotations, distinctions and rules of combination employed. (Chandler) • Benefits: • Provides more insight, captures more compatible data, at lower cost • More amendable to cross-cultural analysis • More accessible/understandable to stakeholders • Much previous data and analysis examples exist • Low overlap with heuristic evaluation
  • 8. Heuristic Evaluation: Criteria • Aesthetic integrity and minimalist design • Consistency and standards • Direct manipulation/See and point • Error prevention • Feedback/Visible system status • Fitt’s Law • Flexibility and efficiency of use • Help and documentation • Help users recognize, diagnose, and recover from errors • Information legibility/Density • Match between system and real world • Modelessness • Perceived stability • Recognition rather than recall • User control and freedom • Visible interfaces/WYSIWYG
  • 9. Semiotic Analysis (SA) • Semiotic analysis focuses on signs in the UI, extracts underlying codes that make/break UI’s meaning. • Primary objective: • Establish underlying conventions • Identify significant differences and oppositions • Model system categories, relations, connotations, distinctions and rules of combination employed. (Chandler) • Benefits: • Provides more insight, captures more compatible data, at lower cost • More amendable to cross-cultural analysis • More accessible/understandable to stakeholders • Much previous data and analysis examples exist • Low overlap with heuristic evaluation
  • 10. Semiotic Analysis: Heuristics • Actors, audience and paradigm. Defining who the UI users are, who the intended audience of the UI is and what the leading interaction paradigm is. • Symbols. Different kinds of symbols connote different semantic spaces, cultural backgrounds and address different audiences. • Syntax. Signs should be used in any given context only once and should not be in conflict with its context. • Rhetorical tropes. The most common rhetorical tropes in the UI are devices of substitution: metaphor, metonymy, prosopopoeia, and synecdoche. • Interaction phases. The interaction should form meaningful temporal units. • Patterns. UI language components form different kinds of patterns.
  • 11. UI Corpus: Selection for Initial Study • To get a workable UI set, study must select sample body of material: UI corpus • Study compared UI of Adobe Photoshop CS2 and GIMP 2.6.7, both running on Mac OS X 10.5.8. • Corpus must be as homogeneous as possible (in substance and time) • Study chose features that were commercially advertised, as well as basic core functions which all users are likely to perform
  • 12. UI Corpus: Functions Analyzed • Study found five shared functions in both UIs: • Reduce red-eye: One-click red-eye tool vs. Red-eye removal • Barrel distortion: Optical lens correction vs. Perspective clone tool • Eliminate an object: Healing brush vs. Clone tool • Clone objects in perspective: Vanishing pt. vs. Perspective clone tool • Customize UI: Keyboard shortcuts and menu customization vs. Configure keyboard shortcuts • Study process analyzed these functions: • Transcribed step-by-step process involved (interaction sentences) • Carried out HE and SA for each UI
  • 13. UI Corpus: Interaction Sentence Example for Adobe Photoshop, 1/2 • (0) Open picture to adjust • (1) Find proper function in menu or tool palette • (a) Sub-task: look in toolbox for button resembling intended action • Nothing like that was found • (b) Alternate sub-task: look through menu items (especially in what seemed most related (Image -> Adjustments, and Filter) for relative command • Found under Filter -> Vanishing point… • Window called “Vanishing Point” appears • Window shows live preview, “Create Plane Tool“, “Clone Tool", etc. • (2) Click four corners according to information provided: • “Click the four corners of a perspective plane or object in the image to create an editing plane. Tear off perpendicular planes from the stretch nodes of existing…)”
  • 14. UI Corpus: Interaction Sentence Example for Adobe Photoshop, 2/2 • (3) Select Clone tool • (4) Option-click in plane to set source • Opt+click in plane to set source point for clone • Once source point is set, click+drag to paint or clone • Shift+click to extend stroke to last click • (5) Click-drag (to paint) several times to clone in perspective • See proposed results in preview. • (6) Click “OK“ to apply changes • (7) Save changes to file
  • 15. UI Corpus: Results of HE vs SA • HE results: • More concise; of 16 heuristics, only small number applied each time • Could be used not only on interaction-sentence level, but also as general analysis of entire UI • SA results: • SA tended to be more verbose • All elements could be applied every time • Could seem repetitive in some instances • Provided solid context of analysis • Enables comparison of UIs from different time-periods to show changes in paradigm, syntax, and related SA criteria
  • 16. Summary • We applied the semiotic and linguistic theories to an UI corpus of graphic manipulation applications. At the same time we analyzed the corpus using heuristic evaluation. • We presented the transcript of interaction sentences forming language games, that served together with the actual UI as a basis for Heuristic evaluation (HE) and Semiotic analysis (SA). Moreover, the transcript served as input for defining the different elements involved in the interaction language. • The SA provided the expected kind of data, that gathered a wider context than those from HE. That said, SA can be used to complement the widely used expert evaluation methods, but could possibly be defined to have a higher overlap with HE. In the latter case, SA would need to be evaluated hand in hand with the interaction sentences. • In summary, our study demonstrated the depth of investigation and breadth of insight that SA can achieve in HCI and how this could enhance the current practice. Both methods could be merged to provide a best-of-both solution.
  • 17. Future Work Plan • Develop a pattern visualization for UI language elements. Such visualization would help quickly compare interaction structures with different kinds of UI in one culture, or between two or more cultures. • Compare the SA with other semiotic methods. • Focus SA to UI pragmatics (persuasion, ideologies).
  • 18. Semiotics of Interaction: Towards a UI Alphabet Jan Brejcha, Doctoral Candidate Charles University in Prague U Kříže 8, 15800 Praha 5 – Jinonice, Czech Republic Tel: +420 602 277 588 Email: Jan@Brejcha.name Web: Jan.Brejcha.name Aaron Marcus, President Aaron Marcus and Associates, Inc. (AM+A) 1196 Euclid Avenue, Suite 1F. Berkeley, California 94708-1640, USA Tel: +1-510-601-0994, Fax: +1-510-527-1994 Email: Aaron.Marcus@AMandA.com Web: www.AMandA.com