This document discusses the history and importance of icons in human-computer interaction. It begins with some of the earliest icons from Orthodox churches in the 6th century. It then covers the development of icons through desktop interfaces in the 1970s-80s and their use in mobile devices like the iPhone. The document emphasizes that icons can bridge the gap between abstract concepts and concrete experiences for users. It also discusses semiotic theories of communication and how they relate to icon design.

1. Icons and the Semiotics of
Human Computer Interaction
Frederick van Amstel @fredvanamstel
Architecture and Design School - PUCPR
www.fredvanamstel.com

2. This is the oldest preserved icon: Christ Pantocrator,
painted in an Orthodox Church (6th Century).

3. This icon probably represented the dual position of
Jesus Christ as both man and god.

4. The contradiction between representation and reality
gave rise to the Byzantine iconoclasm (8th century)

5. The contradiction between representation and reality
intensified in the 20th century (René Magritte, 1928)

6. Representation became oppositional to reality and we
built semiotic machines to manipulate them (1939).

7. Semiotic machines were initially too abstract for those
not trained in Mathematics or Engineering (1964).

8. Icons could bridge the gap by being be both abstract
and concrete, manly as well godly (David Smith, 1975).

9. Desktop metaphor represents abstractions through the
concrete experience of office culture (Tim Mott, 1973).

10. The first computer icons applied to a graphic user
interface (Xerox Alto, 1974).

11. First icon family tested with users (Xerox Star, 1981).

12. Macintosh "iconic" icons made concrete the Personal
Computer concept (Susan Kare, 1984).

13. iPhone: the first product made out of icons (2007).

14. iPhone X: animojis transform facial expressions in icons
in real time (2017).

15. Why are icons so important to
HCI?
• They relate abstract concepts to concrete
experiences
• Mnemonic (easy to memorize and to recognize)
• Easy to find in the screen
• Save screen space
• Internationalize
• Emotional affect

16. Icon anatomy
Border
Label
Background
Predominant color
Lightning
Action
Figure

17. Icon memorization process
Figure shape
and color
Figure and
ground
Distinctive
feature
Complex
image
Icon recognition process

18. Icon meaning: military radar, stealing, vigilance
normalization, worried parents and so on.

19. Semiotic Engineering
• Best known Brazilian
theory of HCI
• Computer as sign-
processing machine
• Interface as a
communication process
based on signs

20. Metacommunication according to Semiotic Engineering
(de Souza, 1993).

21. Operational
metacommunication
(how to use it?)
Strategic
metacommunication
(why using?)
(Barbosa e Silva, 2010)

22. Designer as a translator of programming languages to
interactional languages.

23. Language differences
Programming
• Instructing the
computer
• Code defined by a few
• Formal
• Express computational
concepts
Interactional
• Enabling users to
communicate
• Code defined by many
• Informal
• Express diverse
concepts

24. Interaction design patterns are the basic units of the
interactional language.

25. Icons are interpreted as part of an
interactional language.
Icons are not words.
Icons are sentences.

26. Phrasal structure of an icon: subject (user) verb
(possible action) adverb (action qualifier) predicate
(object) adjective (object qualifier).
Firefox Crystal
Everaldo Coelho
Visual Interactive Syntax Learning

27. Standard iOS Toolbar and Navigation Bar icons have
verbs with implicit predicates.

28. Standard iOS Tab Bar have implicit verbs with multiple
predicates.

29. iOS Home Screen icons predicate have too many
adjectives to express the experience’s qualities.

30. Icon language should employ verbs, nouns and
adjectives in a consistent manner (Spotify).

31. However, formal variation is also important for an icon
language (@MegDraws).

32. Peirce’s triadic sign applied to an icon of a folder with
personal photos.
Object:
Hard disk data
Representamen:
folder in Finder
Interpretant: my
photo album

33. Unlimited semiosis: a sign leads to another sign, which
leads to yet another sign and so on…
Interpretant:
printing the
album
Representamen:
my photo album
Object: my
printed
album
Object:
printed
album gift

34. Semiosis interruption (breakdown): it is not possible to
order a photobook from Photos in Brazil.

35. Communicability expressions to distinguish different
types of breakdowns (De Souza et al, 1999).

36. Semiosis interruption caused by channel noise, unclear
messages or receiver’s lack of repertoire.
designer
user
Where am I? Thanks, but I don’t
want to delete the DVD.

37. In my view, semiosis is, most of the time, interrupted
by the lack of interest or attention.

38. Interest does not come from the looks of the
icon (syntax), not even from the computational
object it represents (semantics), but from what
is possible to do with the icon (pragmatics).

39. The popularity of emojis does not come from the
computational concepts they represent.

40. Icons are increasingly representing non-
computational concepts. They are perceived, as
such, subject to the contradictions of society, in
particular, that one between representation and
reality. Eg: why women should not be represented
in the icon if there are women in the address book?
iOS 10 iOS 11

41. Relationships between representamen and itself: quality,
particularity and Law.
Sinsign
Qualisign
Legisign

42. Relationships between representamen and object:
similarity, causality, abitrary (Nadin, 2017).

43. Relationships between representamen and interpretant:
certainty, fact and possibility (Van Amstel, 2005).
Rheme
Dicent
Argument

44. Representamens can be closer to the object (reality) or
closer to the interpretant (meaning) (McCloud, 1993).

45. Representamen does not
determine the interpretant,
because the object is dynamic.
The sign sometimes represents
this, sometimes that.

46. 1. Define representation parameters before designing
the representamen (Kare, 1982).

47. 2. Generate alternatives to find potential object
representamens (Bigelajzen, 2008).

48. 3. Test with users to verify the relationship between
representamen and interpretant (Van Amstel, 2005).
70% de concordância ou mais 50% de concordância
Recursivas
Hotsites
Mercado
Arquitetura da Informação
Usabilidade
Links
Programação
Consultoria

49. The best way to design a new
icon is not doing it!

50. If there is an icon that reached
the status of a symbol to an
object in a particular culture, it
is wiser to copy it than to
create a new one.

51. Icon libraries
• iconfinder.com
• icons8.com
• flaticon.com
• nounproject.com

52. However, even the most symbolic icons, are subject to
the contradiction of representation and reality.

53. Exercise 1
• Play the Concept game
• Each round should take
no more than 5 minutes
• Discuss the
decomposition of
concepts through
analogies

54. Exercise 2
• Create 10 user stories for an Internet Banking
application using the template:
"As a <type of user>, I want <some goal> so
that <some reason>.
• Make plasticine models for 1) the object and
then add 2) verbs, adverbs and adjectives
• Check their meaning by asking a friend to label
the icons

55. Examples of plasticine icons.

56. Exercise 3
• Sketch an animation to
make clearer the adverb
of the worst icon
• Use the flip animation
technique with a post-it
block
• Add tape to the block's
binding and start from
the bottom sheets

57. Exercise 4
• Sketch many
alternatives for each
icon in your application
• Organize different icons
in rows
• Select the alternatives
that work better
together

58. Thank you!
Frederick van Amstel @fredvanamstel
Architecture and Design School - PUCPR
www.fredvanamstel.com