1. The document discusses how social psychology and behavioral economics have shown that human behaviors are sometimes driven by unconscious decision mechanisms and the perception of social signals unconsciously emitted during interactions. 2. It presents the thin-slice theory, which demonstrates that short snippets of nonverbal behavior can accurately predict important outcomes. 3. The document argues that computer vision and models of human behavior could benefit from incorporating insights about social signaling and seeing people as existing in a social context rather than as isolated individuals. Understanding social signals may enhance artificial perception of human activities and interactions.

1. Bla, bla, bla
Computación y señales sociales
Jordi Vitrià
Universitat de Barcelona
Departament de Matemàtica Aplicada i Anàlisi
&
Centre de Visió per Computador

2. Computación y señales sociales
Pág. 2

3. Computación y señales sociales
Pág. 3

4. Computación y señales sociales
Pág. 4
Index
A new look to human behavior
A new opportunity for machine perception
A new opportunity for machine perception
Social Signals Processing
Social Signals Processing

5. Computación y señales sociales
Pág. 5
A new look to human behavior
Racism and racial segregation.
The thin‐slice theory
y
FROM TWO APPARENTLY NON CONNECTED
FROM TWO APPARENTLY NON‐CONNECTED
TOPICS TO A NEW COMPUTING PARADIGM…

6. Computación y señales sociales
Pág. 6
Thomas C. Schelling,
2005 Economy Nobel Laureate

7. Computación y señales sociales
Pág. 7
Nobel Prize winner Thomas C. Schelling in a 1971
article applied game theory to explain on a
explain,
hypothetical level, how racially mixed neighborhoods
could quickly become segregated even if the people
segregated,
involved are not very segregationist.

8. Computación y señales sociales
Pág. 8

9. Computación y señales sociales
Pág. 9
THE BIRTH OF SOCIAL PHYSICS/NETWORK SCIENCE
Schelling’s game implies that social outcomes needn’t,
at least in some cases, reflect in any obvious way the
l i fl i b i h
desires or intentions, habits or attitudes of anyone
at all.
t ll
His implicit point is that social happenings can have
Hi i li i i i h i lh i h
simple origins and are subject to laws not unlike
those of physics (complexity, network analysis,
small world effects, power laws, etc.)

10. Computación y señales sociales
Pág. 10

11. Computación y señales sociales
Pág. 11
Nalini Ambady & the Thin Slice Theory

12. Computación y señales sociales
Pág. 12
THE EXPERIMENT
In a 1993 study published in the Journal of Personality and Social
Psychology (V l 64 N 3) A b d and a colleague videotaped
P h l (Vol. 64, No. 3), Ambady d ll id t d
13 graduate teaching fellows as they taught their classes.
She then took three random 10‐second clips from each tape,
combined them into one 30‐second clip for each teacher and
showed the silent clips to students who did not know the
teachers. Students assigned a rate to each teacher.
g
THIN SLICE of BEHAVIOR
Then, Ambady correlated that rating with the teachers' end‐of‐
, y g
semester evaluations from actual students.

13. Computación y señales sociales
Pág. 13
THE RESULT
THE RESULT
"We were shocked at how high the correlation
g
was," she says. It was 0.76. In social psychology
anything above 0.6 is considered very strong.
y g y g
These results have been re‐validated in several experiments
and extended to other tasks and environments.

14. Computación y señales sociales
Pág. 14
THIN SLICING and PERCEPTION
What is important is not what we say but
how we behave d i
h b h during the interaction.
h i i
Voice Face Body
Characteristics Expressions Head Postures
Pitch Frowningg
Loudness Head Nod
Head Nod
Smiling Head Shake
Speaking rate Lip‐Pout
Turn taking Head Tilt‐back
Tense‐Mouth …
… …
Body Gestures
y
Gaze Shoulder shrug
Forward lean
Who is she looking at? Hand behind head
Gaze dynamics …
Conjugate lateral eye movement
Fast blink
…

15. Computación y señales sociales
Pág. 15
THIN SLICING THEORY
Thin slices have been shown to have predictive
validity in a number of different contexts:
• teaching (teacher expectancy on students
teaching (teacher expectancy on students,
teacher bias, teacher effectiveness),
• learning (student achievement),
learning (student achievement),
• job performance (telephone operators, sales
managers, management consultants),
managers, management consultants),
• employment interviews,
• health care (doctors effectiveness, patient
health care (doctors effectiveness, patient
satisfaction), etc.

16. Computación y señales sociales
Pág. 16
THIN SLICING THEORY
One of the most impressive examples of thin slices
of data predicting important, long‐term
consequences is marital research conducted by
Gottman and his colleagues.
Carrère and Gottman (1999) were able to predict
marital outcomes over a six year period based on
human social signaling micro‐coding over just the
first 3 minutes of a marital conflict (i.e., a thin slice
of expressive behavior).

17. Computación y señales sociales
Pág. 17
• Importance interest and other social attitudes are reflected in
Importance, interest and other social attitudes are reflected in
speaking style and body language.
• People automatically perceive this information and can use it
p yp
to accurately predict behavior (even being unaware of it!).

18. Computación y señales sociales
Pág. 18

19. Computación y señales sociales
Pág. 19

20. Computación y señales sociales
Pág. 20
Are we taking about emotions?
Are we taking about emotions?

21. Computación y señales sociales
Pág. 21
Paul Ekman
Are we taking about emotions?

22. Computación y señales sociales
Pág. 22
P. Ekman is the most well‐known advocate of this approach, which is based
roughly on the theory that people perceive others’ emotions through
stereotyped displays of facial expression, tone of voice, etc.
The simplicity and perceptual grounding of this theory has recently given rise
to considerable interest in the computational literature .
Are we taking about emotions?

23. Computación y señales sociales
Pág. 23

24. Computación y señales sociales
Pág. 24
Conclusions
1. Social psychology, behavioral economics and other
experimental social sciences h
i l i l i have shown that some
h h
interesting human behaviors are not directly related to
our conscious experience but are driven by rather
experience,
“simple” unconscious decision mechanisms.
2. Moreover,
2 Moreover some of these decision mechanisms are based
on the (unconscious) perception of social signals that are
(
(unconsciously) emitted by all of us when interacting with
y) y g
others.
3.
3 These two approaches defy the classical “cognitive
cognitive
science” approach for understanding human behavior.

25. Computación y señales sociales
Pág. 25
Conclusions
A fundamental assumption of cognitive science is that
the individual is the correct unit of analysis for
understanding human intelligence.
These approaches present evidence that the social
networks containing the individuals are an important
additional unit of analysis, and that this `network
intelligence’ is significantly mediated by non‐linguistic
processes.

26. Computación y señales sociales
Pág. 26
And what about computer
perception of the world?
A new opportunity for machine
A new opportunity for machine
p
perception
p

27. Computación y señales sociales
Pág. 27
The Original Sin of Computer Vision
The Original Sin of Computer Vision
Computer Vision models are based/inspired in
cognitive models from the beginning (Marr & the
theory of reconstruction)
reconstruction).
This models are based on the individual and do not
individual,
consider the social world in which we live.
People is considered just an another “object” that can
be identified tracked etc
identified, tracked, etc.

28. Computación y señales sociales
Pág. 28
One legend in AI: In 1967 Marvin Minsky assigned a graduate student (Gerald Sussman) to the
problem of computer vision, believing that the problem could be largely solved within one
summer.

29. Computación y señales sociales
Pág. 29
The summer project was not successful, but its approach to solve the problem
has not changed a lot from then….
Definition:
As a scientific discipline, computer vision is concerned with the
theory and technology for building artificial systems that obtain
information from images. The image data can take many forms,
such as a video sequence, views from multiple cameras, or multi‐
dimensional d t f
di i l data from a medical scanner.
di l
obtain information from images
=
physical word description

30. Computación y señales sociales
Pág. 30
From David Marr s book: Vision, 1982.
From David Marr's book: Vision 1982

31. Computación y señales sociales
Pág. 31
What about understanding people?
THE CANONICAL VIEW
1. There is a great need for computer programs that can
1 Th i t df t th t
describe and predict people activities from video,
2. This is difficult to do, because it is hard to identify and
This is difficult to do, because it is hard to identify and
track people in video sequences, because we have no
common vocabulary for describing what people are
doing, and because the interpretation of what people
are doing depends very strongly on context.
That’s true, but this is not the whole truth: there is
also a lack of appropriate models for understanding
pp p g
people and their social world.

32. Computación y señales sociales
Pág. 32
Artificial Human Perception is focused on:
Artificial H man Perception is foc sed on
Identity, gender, ethnicity, age, etc.
Emotions
Activities
Gestures (for interaction)
Gestures (for interaction)
But this model of human behavior
is not complete…

33. Computación y señales sociales
Pág. 33
From: A.Vinciarelli, M.Pantic, H.Boulard, Social signal processing: Survey of an emerging domain, Image and Vision Computing, Volume 27, Issue 12, November
2009, Pages 1743‐1759

34. Computación y señales sociales
Pág. 34
Social Signals
Voice Face Body
Characteristics Expressions Head Postures
Pitch Frowning Head Nod
Loudness Smiling Head Shake
Speaking rate Lip‐Pout Head Tilt‐back
… Tense‐Mouth
Tense Mouth …
…
Body Gestures
Gaze Shoulder shrug
Forward lean
Forward lean
Who is she looking at? Hand behind head
Gaze dynamics …
Conjugate lateral eye movement
Fast blink
…

35. Computación y señales sociales
Pág. 35
What we know about social signaling?
It’s short term: People need about 30+ seconds of
observation to infer information from it.
It’s important: there are some real social contexts where
human decisions are mainly (but not exclusively) based
human decisions are mainly (but not exclusively) based
on the content conveyed by this specific, non conscious
communication channel between humans.
It’s a collective game: Information is mainly encoded in
the interaction dynamics (voice and gestures = nonverbal
h d ( d b l
cues).
Arms, body, face, eyes...
Turn‐taking, emphasys,

36. Computación y señales sociales
Pág. 36
We can built social signal detectors by analyzing speech and
body movements
movements.
This signals can be aggregated in communicative patterns
(some of them were proposed by Charles Darwin!)
1.
1 Influence patterns (speaking pitch patterns of turn taking
patterns (speaking pitch, patterns of turn taking,
etc.) It is an indicator of dominance.
2. Activity patterns (energy expended in the communication) It
. patterns (energy expended in the communication) It
is an indicator of interest and excitement.
3. Mimicry patterns (reflexive copying of smiles, interjections,
yp py g j
head nodding) It is an indicator of empathy.
4. Consistency (emphasys and timing). It is an indicator of
determination (+) and openess (‐).

37. Computación y señales sociales
Pág. 37
Functions
F i
1. Forming impressions.
2. Expressing emotions.
3. Sending relational messages (power, persuasion,
dominance, deception…)
dominance deception )
4. Managing interactions.
5. Etc.

38. Computación y señales sociales
Pág. 38
Social cues Example social behaviours Tech.
Emotion Personality Status Dominance Persuasion Regulation Rapport Speech analysis Computer vision Biometry
Physical appearance
Height √ √ √ √
Attractiveness √ √ √ √ √ √ √
Body shape √ √ √ √
Gesture and posture
Hand gestures √ √ √ √ √ √ √
Posture √ √ √ √ √ √ √ √ √
Walking √ √ √ √ √
Face and eyes behaviour
Facial
√ √ √ √ √ √ √ √ √
expressions
Gaze behaviour √ √ √ √ √ √ √ √
Focus of
√ √ √ √ √ √ √ √
attention
Vocal behaviour
Prosody √ √ √ √ √ √
Turn taking √ √ √ √ √ √ √
Vocal outbursts √ √ √ √ √ √ √
Silence √ √ √ √
Space and environment
Distance √ √ √ √ √ √
Seating
√ √ √ √
arrangement
From: A.Vinciarelli, M.Pantic, H.Boulard, Social signal processing: Survey of an emerging domain, Image and Vision Computing, Volume 27, Issue 12, November 2009, Pages 1743‐1759

39. Computación y señales sociales
Pág. 39
Social Signal Processing

40. Computación y señales sociales
Pág. 40
How?
H ?
1. Processing real‐time video/audio and detecting and
analyzing social signals.
2. Building rich descriptions from these signals (social
messages).
g )
3. Understanding syntax and semantics of social
messages (social language) by mining examples and
learning to predict.

41. Computación y señales sociales
Pág. 41
Example: Reading
Example: Reading Faces
• Demographics: Identity, age, gender…
• Emotional Expressions
• Lip Reading
• ?

42. Computación y señales sociales
Pág. 42
Reading Faces
People automatically evaluate faces on
multiple trait dimensions and these
dimensions, and these
evaluations predict important social
outcomes, ranging from electoral success
to sentencing decisions.
to sentencing decisions
Aggressiveness, dominance, confidence,
attractiveness, trustworthiness,
attractiveness, trustworthiness,
competence, etc.

43. Computación y señales sociales
Pág. 43
Dominance

44. Computación y señales sociales
Pág. 44
Competence

45. Computación y señales sociales
Pág. 45

46. Computación y señales sociales
Pág. 46
Body Behavior + Non Verbal Speech
Body Behavior + Non Verbal Speech
(MIT)
1. Speed dating prediction.
2. Call center customer
2 Call center customer
satisfaction.
3. One‐minute elevator
3 “One minute elevator
pitch” assessment.
4. Depression monitoring.
4 Depression monitoring
5. Etc.

47. Computación y señales sociales
Pág. 47
In these scenarios,
we can get an
we can get an
outcome
prediction with a
level of accuracy
level of accuracy
between 80 and
90%

48. Computación y señales sociales
Pág. 48
Organization Dynamics:
Other experiments show that it is
p
possible to identify:
y
•Connectors and central people
in a social network
l k
•The boss in an organization
•The leader of a team
•The outcome of negotiations
•The degree of persuasiveness in
speech Automatically captured group
•Group affiliations
p dynamics

49. Computación y señales sociales
Pág. 49
Long term behavior analysis

50. Computación y señales sociales
Pág. 50
Long term behavior analysis

51. Computación y señales sociales
Pág. 51
Agitation in ICU

52. Computación y señales sociales
Pág. 52
Dominance in computer mediated discussions
 Data
 Blogging heads New
gg g
York Times data base
(http://video.nytimes.com/)
 Experiments
 Observers inquiry
 Manual test
 Automatic test
 Signals
Si l
 Head nodding
 Speaking time (A/V)
 SSuccessful f l
interruptions
 Floor grabbing
 Gesticulation
i l i
52

53. Computación y señales sociales
Pág. 53
Social Signals and Reality Shows
Can we predict who will be fired?
Can we predict who will be fired?
:
The Apprentice is a reality television show that originated in the United
States on NBC. Billed as "The Ultimate Job Interview," the show depicted a
group of 15‐18 businessmen and ‐women competing in an elimination‐style
competition for a one‐year, $250,000 job of running one of host and
executive producer Donald Trump's companies.

54. Computación y señales sociales
Pág. 54
Conclusions
• Social signal processing can contribute to the understanding of the
role of non‐verbal behaviour during social interactions (a new tool
for social sciences).
• The range of application areas for socially aware systems touches
on many aspects of computing, and as computing becomes more
ubiquitous, practically every aspect of i
bi i i ll f interaction with objects,
i i h bj
and the environment, as well as human‐human interaction will
make use of these techniques (situated communication support
q ( pp
tools, human‐computer interaction tools, intelligent tutoring
systems, telecommunication facilities, intelligent multimedia
servers,
ser ers health monitorin s stems intelli ent robots )
monitoring systems, intelligent

55. Bla, bla, bla
Muchas Gracias